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EVALUATION 


The  cost  of  producing  software  has  increased  at  a rapid  pace,  and, 
as  a result,  much  has  been  said  about  the  unreliability  problems 
associated  with  producing  this  software.  The  need  by  the  software 
industry  in  general,  and  the  military  establishment  in  particular, 
to  produce  reliable,  maintainable  and  quality  software,  is  ever 
increasing.  Investigations  into  the  types  of  software  errors  and 
the  reasons  for  their  occurrence  are  taking  place,  but  are  some- 
what deterred  by  an  insufficient  quantity  of  error  data.  The 
error  data  base  can  be  utilized  for  in-depth  analysis  as  well  as 
testing  software  error  prediction  models. 

In  an  attempt  to  overcome  this  insufficient  data  base  and  to 
respond  to  these  needs,  this  effort  was  initiated.  The  effort 
is  in  accord  with  the  goals  of  RADC  TPO  No.  5,  Software  Cost 
Reduction  (formerly  RADC  TPO  No.  11,  Software  Sciences  Technology); 
particularly  in  the  area  of  Software  Quality  (Software  Data).  The 
report  provides  a description  of  the  delivered  data.  The  data 
consists  of  a history  of  software  modifications  to  an  on-board 
flight  software  project  for  a specific  period.  The  value  of 
acquiring  this  data  is  that  it  will  be  analyzed  for  the  purpose 
of  developing  software  measurements  and  will  also  be  used  to  support 
current  software  model  development  projects.  In  addition,  this 
data  will  be  used  concurrently  with  other  procured  software  error 
data,  to  aid  in  establishing  a baseline  for  on-board  flight  soft- 
ware projects  in  quantitative  terms.  This  class  of  information 
will,  in  the  future,  influence  better  methods  of  developing  on- 
board flight  software  projects. 
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SECTION  1 
INTRODUCTION 

This  study  is  based  on  a unique  experience:  Project  Apollo, 
the  manned  lunar  landing,  an  effort  combining  many  technologi- 
cal disciplines.  The  Instrumentation  Laboratory  of  The  Massa- 
chusetts Institute  of  Technology  (MIT/IL) , since  incorporated 
as  The  Charles  Stark  Draper  Laboratory,  Inc.  (CSDL),  was  given 
responsibility  by  NASA  for  the  guidance,  navigation  and  control 
( GNC ) functions  of  the  Apollo  space  vehicles.  The  data  investi- 
gated by  this  study  are  derived  from  the  software  developed  for 
the  computers  of  the  onboard  Primary  GNC  Systems  (PGNCS)  of  both 
the  Command  Module  (CM)  and  Lunar  Module  (LM). 

This  study  compiles  and  categorizes  modifications  that  were 
made  to  the  flight  software  programs  during  the  development  of 
the  PGNCS  of  both  CM  and  LM . This  material  was  recorded  on  mag- 
netic tape;  the  data  contents  are  described  in  Section  4 of  this 
report.  The  purpo^-  of  this  effort  is  to  contribute  to  the  est- 
ablishment of  a are  error  data  base  to  support  research  in 

software  reliab 

A great  de  support  software  was  produced  at  the  labora- 

tory for  the  Apollo  project  in  addition  to  the  flight  software: 
simulators,  assemblers,  data  management  systems,  post  proces- 
sors, and  engineering  simulators.  Figures  for  computer  usage 
and  staffing  include  this  total  software  effort;  the  modifi- 
cation data  collected  and  described  in  this  report,  however, 
are  only  from  the  flight  programs.  These  data  are  compiled 
from  changes  made  to  the  flight  software  for  Apollo  flights  7 
through  17.  Apollo  flights  16  and  17  used  the  same  software  as 
Apollo  15.  The  data  were  collected  from  flight  software  devel- 
oped over  the  years  1967  to  1971. 

This  report  provides  a description  of  the  project  used  as 
the  source  for  these  data,  and  describes  the  data  base  itself. 
Section  2 of  this  report  describes  the  flight  computer  archi- 
tecture, the  functional  nature  of  the  software,  its  production, 
testing  and  management.  Section  3 discusses  the  life  cycle  of 
the  software.  Section  4 describes  in  detail  the  collected  data 
and  its  format.  Section  5 summarizes  the  data  in  graphic  and 
tabular  form.  Section  6 presents  some  conclusions  and  recommen- 
dations drawn  from  this  activity. 
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SECTION  2 
BACKGROUND 

2.1  FLIGHT  COMPUTER  ARCHITECTURE 

The  Apollo  Guidance  Computet  designed  and  developed  by  the 
MIT/I L was  advanced  for  its  day,  being  small,  light-weight  and 
highly  reliable-  During  all  the  hours  of  testing  and  actual 
flight,  not  one  failure  was  ever  recorded.  The  AGC  was  used 
throughout  all  the  Apollo,  Skylab,  Apollo-Soyuz  and  F-8  Phase  I 
programs. 

2.1.1  MEMORY  ORGANIZATION 

The  computer  consisted  of  a fixed  wired  program  memory 
(called  a "core  rope*")  of  36,864  words  in  36  banks  and  a read- 
write  ("erasable")  memory  of  2048  words  in  8 banks.  Words  were 
15  bits  plus  parity;  the  memory  cycle  time  was  12  micro-seconds. 
Fixed  memory  could  not  be  changed  after  manufacture. 

There  were  34  possible  machine  instructions.  Since  15 
bits  were  not  sufficient  to  specify  an  op-code  and  all  38,912 
memory  addresses,  computer  core  was  divided  into  banks  and  pro- 
grammable bank  selection  registers  were  provided  in  the  CPU. 
Any  instruction  could  specify  any  address  within  its  own  bank 
and  could  also  address  the  first  3 banks  of  erasable  as  well  as 
the  first  2 banks  of  fixed  memory.  Access  to  any  other  bank 
was  accomplished  by  using  the  bank  registers.  The  limited  size 
of  erasable  memory  forced  the  time-sharing  of  these  locations. 
Many  software  modifications  resulted  from  this  memory  organiza- 
tion and  appear  in  such  error  categories  as  V050,  program  memory 
optimization,  N010,  items  in  wrong  location,  F040,  organization 
problem. 

Throughout  all  Apollo  programs,  there  was  a "memory 
crunch".  Extreme  efforts  were  expended  by  programmers  to  be 
"clever"  in  order  to  save  even  a few  memory  locations.  Not 
surprisingly,  this  added  to  the  difficulty  of  the  debugging 
process  and  made  ongoing  program  modification  a tricky  business. 
In  some  instances  modifications  were  not  made  even  though 
potential  problems  were  known  to  exist  because  it  was  felt  that 
known  problems  were  less  hazardous  than  new  problems  that  could 
be  introduced  by  a complex  correction-  Care  was  taken  that  the 
problems  were  we)l  understood  and  work-around  procedures  were 
developed,  when  reauired,  to  avoid  them. 


* The  term  "rope"  came  to  be  used  by  Apollo  engineers  to  denote 
each  program  intended  for  release,  even  while  it  was  under 
development.  "Rope"  is  also  used  elsewhere  in  this  report  in 
that  sense. 
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The  fact  that  the  program  memory  was  hard-wired  led  to  an 
effort  (not  covered  by  this  study)  which  extended  through  the 
late  Apollo  flights,  Skylab,  and  the  Apollo-Soyuz  programs.  It 
consisted  of  the  development  of  Erasable  Memory  Programs  (EMPs) 
which  were  implemented  in  order  to  provide  modifications  to  a 
flight  program  while  avoiding  remanufacture  of  the  "rope".  The 
EMPs  resided  in  erasable  memory,  time-sharing  even  further  that 
scarce  resource.  Needless  to  say,  extreme  caution  was  used  in 
specifying,  designing,  implementing  and  testing  each  EMP.  An 
experienced  team  of  experts  participated  in  each  phase  of  EMP 
development,  and  the  astronauts  and  ground  crews  were 

thoroughly  briefed  on  the  use  and  limitations  of  each  EMP. 

2.1.2  INTERRUPT  SYSTEM 

Ten  interrupt  levels  were  provided.  They  are  shown  in 

Table  2-1.  Of  these  ten,  four  were  programmable  counters  of 
varying  granularity,  the  finest  being  10  milliseconds.  These 
counters  were  utilized  to  provide  cyclic  servicing  of  the  vehi- 
cle control  functions,  hardware  calibration  and  other  servicing 
such  as  display  refresh  and  telemetry. 

One  of  the  counters  was  used  to  service  a time  queue  by 
which  the  operating  system  dispatched  asynchronously  timed 
tasks  to  be  processed  in  the  interrupt  mode.  Processing  in  the 
interrupt  mode  was  constrained  to  a time  limit  of  14  milli- 
seconds, since  other  interrupts  were  masked  while  the  current 
interrupt  was  being  processed.  Computer  hardware  failure  moni- 
tors were  provided  and  are  described  in  Table  2-2.  Any  of  these 

failures  caused  a GOPROG  interrupt  and  a subsequent  software 

restart. 

2.1.3  HARDWARE  INTERFACES 

The  computers  were  interfaced  to  the  several  hardware  com- 
ponents shown  in  Figures  2-1  and  2-2. 

All  input/output  (I/O)  took  place  through  counters  and 
channels.  Counters  were  used  for  the  transmission  and  reception 
of  numeric  data;  channels  were  used  for  the  communication  of 
discrete  data.  Counters  were  accessed  by  direct  memory  access 
on  a cycle-steal  basis. 

2.2  FLIGHT  SOFTWARE  FUNCTIONS 

2.2.1  MISSION  PROGRAMS 

The  purpose  of  the  AGC  was  to  compute  guidance,  targeting, 
navigation,  and  control  functions  for  the  Apollo  space  vehicles 
for  all  mission  phases.  Many  guidance,  targeting  and  naviga- 
tion functions  were  computed  by  the  ground  control  system  and 
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Table  2-1  INTERRUPT  STRUCTURE 


INTERRUPT 

PURPOSE 

HIERARCHY 

GOPROG 

RESTARTS 

1 

T6RUPT 

JET  TURN  ON/OFF 

2 

T5RUPT 

DAP 

3 

T3RUPT 

WAITLIST 

4 

T4RUPT 

IMU/OPTICS  MONITORING 

5 

KEYRUPT  (2  ) 

KEYSTROKE  PROCESSING 

6,7 

UPRUPT 

UPTELEMETRY  PROCESSING 

8 

DOWNRUPT 

DOWNLINK  PROCESSING 

9 

RADRUPT 

RADAR  RETURN  PROCESSING 

10 

Table  2-2  COMPUTER  HARDWARE  INTERRUPTS 


OSCILLATOR  FAIL 

Occurs  if  loss  of  oscillator  1.02  MHz  square  wave 
happens.  In  addition  a logic  circuit  insures  a 
RESTART  condition  for  a 250  millisecond  interval  upon 
transferring  from  STANDBY  to  OPERATE. 


TRANSFER  CONTROL  (TC)  TRAP 

Occurs  if  too  many  or  too  few  TC  instructions  are 
requested.  The  period  for  "too  many"  or  "too  few"  is 
from  5 to  15  milliseconds  in  duration. 


PARITY  ALARM 

Occurs  if  any  accessed  word  in  fixed  or  erasable 
memory  whose  address  is  10  octal  or  greater  contains 
an  even  parity  of  "ones."  All  locations  of  10  octal 
or  greater  are  stored  in  fixed  or  erasable  memory  with 
odd  parity. 


NIGHTWATCHMAN  FAIL 

Occurs  if  the  computer  should  fail  to  access  address 
67  within  a period  whose  duration  varies  from  .64  to 
1.92  seconds.  This  assures  that  the  computer  is  still 
operating  during  an  extended  idle  period  and  is  tied 
up  in  an  interrupt  loop. 


INTERRUPT  (RUPT)  LOCK 

Occurs  if  an  interrupt  is  either  "too  long"  or  "too 
infrequent"  • 


VOLTAGE  FAIL 

Occurs  if  the  AGC  voltages  are  out  of  limits  for  157 
to  470  microseconds. 


IMAND  MODULE  GUIDANCE,  NAVIGATION  AND  CONTROL 


RENDEZVOUS 
RADAR  ✓ 


Figure  2-2  LUNAR  MODULE  GUIDANC 


duplicated  on-board,  but  control  functions  were,  of  course,  an 
on-board  responsibility;  so  too  were  computations  of  range, 
range-rate,  velocity  requirements,  and  other  parameters  for  lun- 
ar landing  and  rendezvous-  These  computations  were  based  on 
radar  and  optics  sensor  inputs  and  monitored  by  the  astronauts, 
who  had  final  authority  over  all  activity.  The  following  defi- 
nitions of  these  functions  appear  in  reference  1- 

Navigation  - the  measurement  and  computation  necessary 
to  determine  the  present  spacecraft  position  and  velo- 
city. 

Targeting  - the  computation  of  the  maneuver  required 
to  continue  to  the  next  step  in  the  mission. 

Guidance  - the  continuous  measurement  and  computation 
during  accelerated  flight  to  generate  steering  signals 
necessary  to  assure  that  the  position  and  velocity 
changes  of  the  maneuver  will  be  those  required  by  navi- 
gation measurements  and  targeting  computations. 

Control  - the  management  of  spacecraft  attitude  motion; 
the  rotation  to  and  maintenance  of  the  desired  space- 
craft attitude  during  free-fall  coasting  flight  and 
powered  accelerated  flight- 

Table  2-3  lists  the  major  mission  programs  that  were  available 
for  astronaut  selection. 

2.2.2  CREW  INTERFACE 

Because  of  the  importance  of  the  man-machine  interface,  a 
significant  portion  of  the  software  effort  was  devoted  to  dis- 
plays and  handling  of  crew  keyboard  inputs.  At  significant 
points  in  the  processing,  a display  was  presented  to  the  crew; 
the  program  did  not  advance  further  until  directed  to  do  so  by 
the  crew  (depression  of  a PROCEED  key  was  the  standard  method 
of  crew  approval). 

These  paragraphs  are  adapted  from  material  presented  in 
reference  1. 

The  basic  man/computer  interface  device  is  the 
display  keyboard  ( DSKY ) (shown  in  Figure  2-3).  Through 
the  DSKY  the  astronaut  could  initiate,  monitor,  or 
change  programs  being  processed  by  the  computer.  He 
could  request  the  display  of  specific  data  or  enter  new 
data.  Communication  with  the  DSKY  was  two-way*.  the 
astronaut  could  exercise  command  via  the  DSKY  and  the 
computer  could  request  the  astronaut  to  monitor, 

approve,  or  enter  data  when  necessary.  There  were  two 
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Table  2-3  MISSION  PROGRAMS 


COMMAND  MODULE 


CMC  Idling 

Prelaunch  Initialization 
Gyro  Compassing 
Verify  Gyro  compassing 
CMC  Power  Down 
IMU  Ground  Test 
EARTH  ORBIT  INSERTION 
Earth  Orbit  Insertion 
NAVI GATION 

Rendezvous  Navigation 
Ground  Track  Determination 
Orbital  Navigation 
Cislunar  Navigation 
CMC  Update 

RENDEZVOUS  TARGETING 
External  Delta  V 
Lambert  Aimpoint 
Coelliptic  Sequence 
Constant  Delta  Height 
Transfer  Phase  Init 
Transfer  Phase  Midcourse 
Return  to  Earth 
POWERED  FLIGHT 
SPS  Maneuver 
RCS  Maneuver 
IMU  ALIGNMENT 
IMU  Orientation 
IMU  Realign 

Backup  IMU  Orientation 
Backup  IMU  Realign 
ENTRY 

Maneuver  and  Separation 
Separation  and  Reentry 
Entry-Initiation 
Entry-Post  -05  g 
Entry-Up  Control 
Entry-Ballistic 
Entry-Final  Phase 
LM  RENDEZVOUS  TARGETING 
LM  Coelliptic  Sequence 
LM  Constant  Delta  Height 
LM  TP  I 
LM  TPM 

Target  Delta  V 


LUNAR  MODULE 


00  LGC  Idling 

06  GNCS  Power  Down 
ASCENT 

12  Ascent  Guidance 
NAVIGATION 

20  Rendezvous  Navigation 

21  Ground  Track 

22  Surface  Navigation 

25  Preferred  Tracking  Attitude 
27  LGC  Update 
RENDEZVOUS 

30  External  Delta  V 

31  Lambert  Aimpoint  Maneuver 

32  Coelliptic  Sequence 

33  Constant  Delta  Height 

34  Transfer  Phase  Initiation 

35  Transfer  Phase  Midcourse 
POWERED  FLIGHT 

40  DPS  Maneuver 

41  RCS  Maneuver 

42  APS  Maneuver 
47  Thrust  Monitor 

IMU  ALIGNMENT 

51  IMU  Orientation 

52  IMU  Realign 

57  Lunar  Surface  Align 
LANDING 

63  Braking  Phase 

64  Approach  Phase 

65  Landing  Phase  (auto) 

66  Landing  Phase  (ROD) 

67  Landing  Phase  (manual) 

68  Landing  Confirmation 
BACKUP 

70  DPS  Abort 

71  APS  Abort 

72  CSM  CSI  Targeting 

73  CSM  CDH  Targeting 

7 4 CSM  TP  I Targeting 

75  CSM  TPM  Targeting 

76  Target  Delta  V 
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DSKYs  available  in  the  CM  and  one  in  the  LM.  Each  DSKY 
had  a keyboard,  several  electro-  luminescent  displays, 
and  activity  and  alarm  lights.  The  activity  lights 
were  for  the  computer  and  the  telemetry  uplink,  and 
the  alarm  lights  were  for  the  computer  and  inertial 
subsystems.  These  aided  the  astronaut  in  monitoring 
the  status  of  the  G&N  system.  The  alarm  lights  indi- 
cated equipment  failure  and  program  alarms. 

The  basic  language  used  for  communication  between 
the  operator  and  the  computer  was  a pair  of  two 
character  numbers  that  represented  a verb/noun  combin- 
ation. The  verb  code  indicated  the  operation  to  be 
performed,  while  the  noun  indicated  the  operand  to 
which  the  operation  (verb)  applied.  Typical  of  the 
verb  codes  used  are  those  for  displaying  and  loading 
data.  For  example,  VERB  16  NOUN  20  provided  a 
monitored  display  of  gimbal  angles  and  VERB  25  NOUN  18 
loaded  the  desired  gimbal  angles. 

Noun  codes  called  up  groups  of  erasable  registers 
within  computer  memory.  Processing  of  nouns  provided 
information  in  units  scaled  for  ease  of  crew  use. 

The  users  of  the  system,  the  Apollo  astronauts,  were  very 
much  involved  in  the  design  of  the  software/crew  interface.  In 
the  early  flights,  a great  deal  of  keyboard  activity  was 
required  of  the  astronauts  who  wanted  total  control  over  the 
selection  of  computer  program  sequences-  Later  in  the  Apollo 
program,  as  confidence  in  the  computer  system  increased,  the 
crews  were  willing  to  relinquish  some  of  that  authority  and  the 
program  sequences  were  automated  to  a greater  extent.  One  sig- 
nificant effort  in  that  direction  was  the  development  of  the 
MINKEY  (for  minimum  keystroke)  sequence  in  the  command  module 
rendezvous  program.  The  MINKEY  sequence  automated  what  was 
formerly  a burdensome  period  for  the  single  astronaut  aboard 
the  command  module.  Another  effort  that  was  undertaken  after 
the  first  manned  flights  was  the  incorporation  into  the  soft- 
ware of  the  error  checking  of  all  crew  inputs.  Experience  had 
shown  that  illegal  or  incorrect  inputs  caused  aberrant  behavior 
and  could  have  seriouslv  impacted  mission  performance.  Legal- 
ity checks  in  the  software,  although  consuming  scarce  memory 
resources,  were  deemed  necessary  to  guard  against  a potentially 
dangerous  source  of  error. 

2.2.3  RESTARTABI LITY 

A unique  aspect  of  the  Apollo  software  was  its  built-in, 
real-time  error  recovery  and  restart  capability.  This  capability 
was  implemented  by  storing  restart  pointers  at  strategic  points 
in  each  process.  The  pointers  were  stored  in  "restart  tables", 
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which  accommodated  up  to  six  separate  processes  at  any  time. 
Determination  of  restart  points  was  a complex  process  based  on 
the  repeatability  of  coding  sequences.  Updating  of  variables 
was  protected  by  storing  a copy  of  the  updated  value,  inserting 
a restart  point,  then  replacing  the  old  values  with  the  newly 
computed  ones.  Other  coding  sequences  posed  more  difficult 
logic  problems  for  the  programmers,  sometimes  necessitating 
multiple  restart  points  within  a few  lines  of  code-  The  multi- 
programmed  nature  of  the  software  contributed  to  the  complexity 
of  the  restart  design-  The  restart  tables  were  time-shared,  so 
that  the  assignment  of  table  locations  to  various  processes 
required  expert  knowledge  of  the  total  program  activity 
throughout  the  mission.  Also,  insertion  of  restart  points  in 

the  code  had  to  take  into  consideration  the  possibility  that 
processes  could  be  preempted  by  a higher  priority  process  at 
almost  any  time.  Although  -this  capability  was  expensive  in 
terms  of  memory  usage,  as  well  as  being  a significant  source  of 
errors  during  development,  this  feature  was  directly 

responsible  for  the  success  of  the  Apollo  11  and  12  missions, 
which  may  well  have  failed  without  it-* 

Modifications  to  the  software  for  the  implementation  of  this 
capability  are  recorded  under  categories  H040,  inadequate 
restart,  or  H050,  errors  in  restart  logic. 

2.2.4  MULTIPROGRAMMING  AND  ASYNCHRONISM 

The  AGC  operating  system  was  designed  and  implemented  very 
early  in  the  program,  before  the  time  period  covered  by  this 
study.  It  provided  great  flexibility  in  that  it  allowed  for  both 
synchronous,  precisely  timed  cyclic  processing  and  asynchronous 
tasks  which  could  themselves  be  either  cyclic,  timed  processes 
or  priority-driven  jobs.  Timed  processing  was  interrupt-driven 
and  was  itself  non-interruptable.  For  this  reason  timed  proces- 
sing was  subject  to  a time  limit  of  14  milliseconds.  Typical 
synchronous  timed  processes  were  the  digital  autopilot  (DAP) 
and  the  hardware  monitoring  and  service  routines.  Dedicated 
interrupts  were  assigned  to  these  processes. 


*Lightning ' struck  the  Apollo  12  vehicle  during  the  ascent  phase 
and  caused  a series  of  power  transients  in  the  computer 
system.  The  software  recovery  system  successfully  restarted 
the  program  allowing  the  mission  to  continue.  The  recovery 
system  was  similarly  involved  in  the  lunar  landing  phase  of 
Apollo  11  when  erroneous  radar  inputs  caused  a computational 
overload.  The  recovery  system  deleted  low  priority  functions, 
performing  only  essential  computa tional  sequences,  and  thus 
relieving  the  computer  overload  and  allowing  successful  comple- 
tion of  the  landing. 
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Asynchronously  timed  processes  were  dispatched  by  the  oper- 
ating system  from  a time  queue  (called  WAITLIST)  to  which 
another  interrupt  was  dedicated.  These  tasks  were  scheduled 
in  response  to  real-time  program  requirements. 

Priority-driven  jobs  were  also  scheduled  as  required  by  cur- 
rent demands  on  the  system,  often  in  direct  response  to  crew 
inputs.  These  jobs  were  preemptable;  that  is,  they  could  be 
interrupted  by  timed  tasks  (unless  interrupts  were  specifically 
masked)  or  by  jobs  of  higher  priority. 

Interaction  between  processes  in  real  time,  the  sharing  of 
resources  necessitated  by  limited  memory,  and  the  fact  that 
processes  could  be  initiated  asynchronously  and  unpr edictably, 
created  a complexity  in  program  behavior  that  reauired  extensive 
testing  and  often  heroic  debugging  efforts.  The  Assembly  Cont- 
rol Supervisor  (para  2.4.1)  and  a relatively  small  group  of 
software  experts  were  often  called  upon  to  dig  into  a problem 
to  help  explain  the  seemingly  inexplicable. 

2.3  FLIGHT  SOFTWARE  PRODUCTION  AND  TESTING 

The  laboratory  began  its  Apollo  work  in  1961,  and  the 
early  years  were  primarily  devoted  to  hardware  development, 
including  computer  design  and  prototyping.  Early  software  work 
was  performed  by  a small  group  of  engineers  who  were  familiar 
with  the  computer  design  work  and  who  had  been  closely  involved 
with  the  working  groups  that  developed  the  mission  requirements. 
All  of  the  programs  for  the  manned  Apollo  flights  were  based  on 
this  initial  existing  "core"  of  system  (e.g.,  executive,  display 
and  hardware  interface)  software.  Thus  each  new  program  was 
essentially  a modification,  although  usually  a large-scale  modi- 
fication, of  a previous  release.  With  the  approach  of  manned 
missions,  software  requirements  grew  and  the  programming  and 
verification  group  was  expanded.  Figure  2-4  shows  how  this 
effort  was  staffed  over  the  years. 

2-3.1  PRODUCTION 

The  group  responsible  for  the  flight  software  included 
guidance,  navigation,  and  control  engineers,  programmers,  and 
test  engineers,  led  by  a small  team  who  had  been  associated 
with  the  early  work  at  the  laboratory  and  who  had  themselves 
developed  the  first  test  and  flight  programs.  Experts  in  the 
various  disciplines  worked  closely  together  and  in  many  cases  a 
single  person  participated  in  all  phases  of  development  of  a 
particular  software  module,  i-e.,  initial  engineering  studies, 
programming  and  testing- 

The  coding  was  done  both  in  the  assembly  language  of  the  AGC 
and  in  the  interpretive  language  (INTERPRETER)  developed  for  the 
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project.  Fixed  point  arithmetic  was  used  throughout-  Most 
mathematical  programming  was  done  in  INTERPRETER,  which  provided 
pseudo  instructions  for  matrix  and  vector  arithmetic,  trigono- 
metric functions,  and  double  precision  arithmetic.  The  code 
was  assembled  by  a cross-assembler  hosted  first  on  the  Honeywell 
1800  computer  and  later  on  the  IBM  360/75. 

2.3.2  TESTING 

2. 3. 2-1  TYPES  OF  FACILITIES 

Testing  and  verification  at  the  laboratory  were  performed 
using  various  facilities,  including  engineering  simulation  on 
the  host  computer,  full  scale  digital  simulation  on  the  host 
computer,  and  a hybrid  laboratory  and  system  test  laboratory 
chat  provided  real-time  execution.  Engineering  simulations 
were  primarily  used  as  a design  tool  to  test  algorithms  and 
techniques  before  actual  incorporation  into  the  flight  program. 
Some  of  these  simulations  were  used  throughout  the  Apollo 
program  to  evaluate  the  performance  of  the  mission  programs  and 
pr ocedur es. 

By  far  the  largest  amount  of  in-house  code  verification  was 
performed  using  the  all-digital  simulator,  a sophisticated  tool 
which  performed  high-fidelity  environment  modelling  and  inter- 
pretive simulation  of  the  AGC  and  provided  a large  variety  of 
diagnostic  tools  and  user  options. 

The  hybrid  laboratory  consisted  of  2 complete  simulators, 
one  for  the  lunar  module  and  one  for  the  command  module.  Mock- 
ups  of  the  CM  and  LM  cockpits  were  interfaced  with  the  hybrid 
computers  to  provide  a realistic  environment.  A XDS-9300 
computer  controlled  the  simulation.  It  enabled  real-time  test- 
ing of  the  crew  interfaces  and  procedures  with  the  flight  soft- 
ware (in  a core  rope  simulator)  and  a mix  of  real  and  simulated 
hardware. 

The  system  test  laboratory  also  contained  two  complete  hard- 
ware systems,  one  for  the  LM  and  one  for  the  CM.  It  provided  a 
test  bed  which  included  a simulated  guidance  computer  (the  core 
rope  simulator),  actual  radar,  optics,  and  inertial  measurement 
units. 

The  hybrid  and  system  test  laboratories  were  extensively 
used,  in  parallel  with  digital  simulation,  for  level  3,4,5  and 
6 tests  (see  below).  Levels  1 and  2 were  performed  exclusively 
on  the  digital  or  engineering  simulators. 
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2. 3. 2. 2 LEVELS  OF  TESTING 


Several  levels  of  testing  were  performed. 

Level  1 rests  were  high  order  language  (HOL)  programs  run  on 
the  host  computer  to  test  algorithms.  The  MAC  (MIT  Algebraic 
Compiler),  developed  at  MIT/IL,  was  used  for  this  effort. 

Level  2 was  the  AGC  counterpart  of  these  programs-  The 
results  of  the  two  were  compared  to  establish  the  accuracy  of 
the  AGC  equivalent-  The  errors  found  at  this  level  were  prim- 
arily computational - 

Level  3 was  intended  to  verify  the  operation  of  a complete 
program  or  routine  including  crew  interface  and  realistic  physi- 
cal environment  models.  The  errors  discovered  at  this  level 
were  primarily  logic  and  display  interface  problems.  This 
level  was  performed  only  when  a routine  was  incorporated  into 
the  flight  program. 

Level  4 testing  was  intended  to  verify  mission  phases, 
e-g.,  ascent,  rendezvous.  The  multi-programmed  environment  was 
exercised  extensively  and  therefore  uncovered  priority,  timing, 
and  erasable-sharing  problems. 

Level  5 repeated  the  level  4 tests  on  the  final  rope  which 
was  released  for  manufacture.  This  was  required  because  even 
chough  che  level  4 tests  had  been  successfully  completed,  they 
may  not  have  run  on  che  version  of  the  program  that  was 
released • 

Level  6 cook  place  afcer  che  ropes  were  released  for  manu- 
faccure  and  were  incended  co  verify  che  program  using  accual 
mission  daca  and  che  flighc  time-line.  These  runs  were  run 
wich  1 sigma  and  3 sigma  errors  in  the  simulated  instruments. 


2.4  IN-HOUSE  CONFIGURATION  CONTROL 

The  very  early  programs,  produced  by  a small  cadre  of 
dedicated  engineers,  were  not  subject  to  formal  configuration 
control  procedures.  By  the  time  of  the  period  covered  by  this 
study,  however,  the  magnitude  of  the  task  and  the  large  number 
of  contributors  necessitated  control  procedures  to  insure  the 
continuing  integrity  of  the  software. 

2.4.1  ASSEMBLY  CONTROL  SUPERVISOR 

The  assembly  control  function  was  established  to  localize 
responsibility  for  tht  quality  of  each  program  as  it  was  being 
developed.  The  Assembly  Control  Supervisor  (ACS)  was  a person 
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who  was  knowledgeable  about  all  aspects  of  the  program,  and 
especially  expert  in  the  system  software,  subprogram  interfaces, 
initialization  and  crew  interfaces.  The  ACS  could  call  on  a 
group  of  "experts"  in  the  various  disciplines  to  act  as  consult- 
ants to  resolve  problems-  All  code  was  submitted  to  the  ACS 
for  approval  before  it  was  incorporated  into  the  official  assem- 
bly. The  code  was  documented  using  coding  forms*  which  provided 
information  about  the  nature  of  the  code,  the  reason  it  was 
being  incorporated,  the  official  change  notice  or  change  request 
reference  {if  any),  and  the  signatures  of  the  programmer  and 
his  supervisor,  which  guaranteed  that  the  code  had  been  checked 
oui  independently  before  it  was  submitted. 

It  was  only  after  carefully  scanning  the  code  and  analyzing 
it  with  respect  to  its  interfaces  with  existing  software  that 
rhe  ACS  approved  each  submittal  for  incorporation  into  the 
official  assembly.  The  ACS  issued  memos  documenting  in  detail 
each  change  that  had  been  included  into  each  new  assembly 
revision. 

2.4.2  ASSEMBLY  CONTROL  BOARD 

The  assembly  control  activity  described  above  was  under  the 
overall  management  of  a system  integration  group.  A board  con- 
sisting of  an  integration  supervisor,  the  ACS  for  each  separate 
program**  and  a group  of  experts  was  established  to  provide  pol- 
icy guidance  and  resolve  technical  issues  not  able  to  be  decided 
by  the  ACS  alone- 

For  testing  and  checkout  before  submittal  to  the  official 
assembly,  engineers  could  snapshot  a version  of  the  assembly, 
incorporate  their  new  code,  run  simulations,  and  modify  and 
remodify  their  programs. 

2.4.3  ERASABLE  COMMITTEE 

This  system,  with  its  built-in  checks  at  several  levels 
{engineer,  supervisor,  ACS,  experts),  worked  well  to  provide 
visibility  and  control  during  development  phases  of  the  project. 
Because  only  2048  words  of  read-write  (erasable)  memory  were 
available,  it  was  necessary  to  time-share  data  memory  among  many 
software  modules-  Assuring  the  integrity  of  erasable  data 

required  an  overview  of  the  program  behavior  and  a knowledge  of 


*These  coding  forms  served  as  the  source  of  the  data  prepared 
for  this  study. 

**  There  were  always  at  least  two  programs  under  development, 
the  LM  and  CM  programs  and  at  times  programs  for  different 
flights. 
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the  timing  characteristics  and  interactions  of  the  various 
program  modules-  To  this  end  a committee  was  established  to 
oversee  the  management  of  erasable  memory;  the  ACS  was  an 
important  member  of  this  committee.  Requests  for  data  storage 
from  engineers  working  on  the  software  were  reviewed  by  the 
committee  which  assigned  data  locations  on  the  basis  of  global 
needs  and  time-sharing  considerations. 


Some  consideration  was  given  to  automating  the  process  of 
erasable  assignments.  Studies  that  were  conducted  at  the  timer 
however,  led  to  the  conclusion  that  the  effort  and  accuracy 
required  to  provide  correct  input  describing  data  requirements 
and  time-sharing  characteristics  would  be  as  difficult  for  the 
automated  process  as  it  was  for  the  manual  activity. 

Figure  2-5  is  a typical  erasable  memory  map  showing  the  data 
overlays  in  one  bank  of  erasable  memory. 


ERASABLE  MEMORY  MAP  EBANK  7 1400-1600 
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SECTION  3 


SOFTWARE  LIFE  CYCLE 


3-1  SYSTEM  SOFTWARE 

Much  Apollo  flight  software  was  developed  before  the  con- 
trols discussed  in  Section  2 were  established-  This  included 
both  system  software  and  mission  software.  Although  the 
mission-related  software  was  eventually  replaced  by  software 
capable  of  performing  the  lunar  landing  mission,  the  system 
software  was  basically  unchanged-  This  software  included  the 
executive,  display  interface,  interpreter,  much  of  the  hardware 
interface  logic,  interrupt  handling  and  computer  self-test-  In 
general,  the  system  design  was  not  formally  documented  and  was 
produced  by  a relatively  small  group  of  engineers  who  were  thor- 
oughly knowledgeable  of  the  performance  requirements  of  the 
hardware,  the  sensor  instruments  and  the  computer.  As  the  pro- 
grams continued  to  develop  and  as  more  and  more  testing  was 
done,  some  changes  were  made  to  these  programs  for  improvement 
and  error  corrections,  but  these  corrections  were  relatively 
few- 

3-2  MISSION  SOFTWARE  REQUIREMENTS 

The  requirements  for  the  Apollo  mission  were  determined 
jointly  by  NASA  and  the  laboratory.  Once  the  lunar  orbit  rendez- 
vous technique  was  established  as  the  method  of  accomplishing 
the  ultimate  goal  of  Apollo,  some  obvious  and  desirable  require- 
ments became  apparent,  e-g-,  navigation,  guidance,  etc.  The 
software  was  designed  to  perform  the  mission  functions  on-board 
even  though  many  functions  were  duplicated  on  the  ground. 

3.3  MISSION  PROGRAM  DEVELOPMENT 

por  the  period  of  time  covered  by  this  study,  software 
development  consisted  of  creating  a program  for  a particular 
mission,  testing  that  program,  and  releasing  it  for  manufacture. 
This  cycle  was  repeated  for  each  release.  The  major  utilization 
of  host  computer  resources  during  a program's  life  cycle  was  in 
testing.  Testing  continued  after  every  release  to  evaluate  any 
deviation  from  the  planned  mission- 

In  the  development  of  software  capabilities,  use  was  made  of 
engineering  simulations  of  varying  complexity  to  validate  the 
algorithms  and  techniques  before  actual  implementation  into  the 
flight  software. 

During  the  development  phase  a data  base  management  system, 
List  Processing  Service,  LIPSVC,  was  used.  The  LIPSVC  system 
provided  reliablity  and  visibility  in  that  earlier  program  re- 
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visions  could  be  recreated  at  any  time;  it  provided  control  of 
the  frequent  assemblies  and  permitted  easy  creation  of  "off- 
line" versions  while  recording  all  modifications  to  these  ver- 
sions. This  allowed  new  programs  to  be  debugged  without  inter- 
fering with  the  main  line  production.  The  interim  modifications 
to  the  off-line  versions  were  not  centrally  maintained  and 
therefore  do  not  appear  in  the  data  collection. 

There  often  was  parallel  development  of  software  for  the 
same  vehicle.  For  example,  Colossus  3,  the  software  program 
which  flew  on  Apollo  flights  15,  16  and  17,  was  begun  by  snap- 

ping a version  of  the  program  named  Comanche  under  the  new 
name,  Artemis.  This  new  version  was  begun  in  parallel  with  the 
Apollo  12  program.  Since  extensive  changes  were  planned  for 
Apollo  15,16  and  17,  a major  effort  was  undertaken  to  recode 
vast  portions  of  the  existing  program  solely  for  the  purpose  of 
gaining  space  for  the  new  features.  Rather  than  waiting  for 
the  release  of  the  Comanche  program  for  these  changes,  the  new 
version  was  developed. 

3.4  SPECIFICATIONS  AND  DOCUMENTATION 


The  controlling  document  in  the  life  cycle  of  the  Apollo 
software  was  the  Guidance  System  Operation  Plan  (GSOP),  a docu- 
ment which  served  as  the  specification  for  the  software  efforts. 
Development  and  control  of  the  GSOP  were  important  activities 
in  planning,  controlling,  and  documenting  a flight  program.  For 
early  flights  the  GSOP  was  a single  volume  document.  With  the 
advent  of  manned  flights,  it  was  expanded  to  six  separate 
volumes.  Each  volume  of  the  GSOP  was  dedicated  to  controlling  a 
different  aspect  of  the  AGC  software.  Section  1 controlled  the 
AGC  prelaunch  activities.  Section  2 dealt  with  data  links, 
uplink,  downlink,  and  telemetry.  Section  3 dealt  exclusively 
with  the  digital  autopilot.  Section  4 governed  operational 
modes  including  PGNCS  interfaces  with  the  flight  crew  and  mis- 
sion control.  Section  5 contained  the  guidance  and  navigation 
equations.  Section  6 specified  the  data  used  in  the  digital 
and  hybrid  simulators  in  support  of  the  verification  of  the  AGC 
programs. 


In  addition  to  the  function  of  the  GSOP  as  a NASA  control 
document,  it  served  as  an  internal  working  document,  as  a test- 
ing guide,  and  as  a crew  training  aid. 
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Changes  to  the  software  or  the  GSOP  were  controlled  by  the 
documents  described  below. 

•Program  Change  Request  (PCR) 

A PCR  was  a request  for  a change  in  the 
specification  (GSOP)  for  a flight  program- 
It  was  given  a preliminary  review  by  a lab- 
oratory engineer  and  by  the  NASA  Flight 
Software  Branch  and  then  held  for  Software 
Control  Board  action.  The  SCB  was  composed 
of  representatives  of  various  branches  of 
NASA.  SCB  could  disapprove  the  change, 
require  more  detailed  evaluation  by  MIT/IL 
or  require  implementation  of  the  change. 

This  decision  involved  overall  mission  con- 
siderations and  scheduling,  as  well  as  the 
particular  software  considerations. 

s 

•Program  Change  Notice  (PCN) 

A PCN  originated  at  MIT/IL  and  was  a noti- 
fication that  a change  was  being  made 
rather  than  a recruest  for  a change.  The  PCN 
was  used  for  clerical  corrections  to  the 
flight  program  specifications  or  for  changes 
that  were  required  to  the  program  for  devel- 
opment to  continue-  PCNs  required  approval 
by  the  SCB,  but  approval  was  usually  auto- 
matic- If  the  SCB  disapproved,  a change  to 
undo  the  PCN  was  generated. 

4 

•Anomaly  Report 

Anomaly  Reports  reported  a discrepancy 
between  a program's  specification  and  its 
operation.  Anomaly  Reports  required  offi- 
cial disposition  ,but  did  not  always  result 
in  program  modification-  Work-around  pro- 
cedures or  EMPs  were  sometimes  developed  to 
correct  the  anomaly.  Table  3-1  shows  the 
number  of  known  anomalies  that  existed  in 
each  flight. 

• Assembly  Control  Board  Request 

An  Assembly  Control  Board  Request  was  pre- 
pared by  MIL/lL's  Assembly  Control  Board 
and  initiated  a program  change-  This  change 
did  not  change  the  program's  specification. 

Instead  it  was  in  the  nature  of  an  in-house 
improvement . 
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3.5  SOFTWARE  RELEASES 

When  the  laboratory  was  directed  to  release  the  mission 
program,  a magnetic  tape  was  produced  for  delivery  to  the  manu- 
facturer. The  manufacturing  process,  which  required  at  least 
45  days,  started  by  processing  the  magnetic  tape  to  produce  2 
mylar  tapes.  One  tape,  the  core  rope  weaver,  actuated  the 
weaving  machines  that  produced  the  module  memory.  The  other 

tape  was  used  to  verify  the  memory  fabrication  process. 

Figure  3-1  shows  the  development  of  the  flight  ropes  for 
the  Command  and  Lunar  Modules. 

3.6  MISSION  SUPPORT 

All  missions  were  supported  around  the  clock  by  laboratory 
personnel  both  in  Cambridge  and  at  NASA  facilities.  Although  no 
software  problems  occured  in  flight,  work-around  procedures 
invoked  through  the  software  were  occasionally  required  to  solve 
GNCS  problems*.  These  procedures  had  to  be  developed  very 
quickly  to  allow  as  much  testing  as  possible  before  they  were 
transmitted  to  the  crew. 


‘Apollo  14  required  a procedure  to  bypass 
LM  abort  switch  by  the  software. 
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checking  the  failed 
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SECTION  4 

APOLLO  SOFTWARE  DATA  FILES 

Two  files  of  da  ta  have  been  delivered  to  the  sponsor  of 
this  study.  The  first  file  contains  11,729  records,  each  of 

which  describes  a modification  to  the  software,  called  a Record 
of  Software  Modification  (RSM).  The  second  file  contains  83 
records,  and  gives  data  on  other  errors,  machine  use  statistics, 
speed  of  simulation  and  charater istics  of  the  software. 

4.1  APOLLO  SOFTWARE  MODIFICATIONS 

The  basic  data  in  the  RSMs  (record  identifier,  date  of  mod- 
ification, revision  identifier,  reference,  functional  category, 
modification  category,  modification  description),  were  prepared 
by  a team  of  14  engineers  with  experience  in  developing  Apollo 
software.  The  rest  of  the  data  in  the  RSMs  was  computationally 
derived  from  this  basic  data. 

Each  RSM  was  prepared  from  one  of  2 types  of  software 
Modification  Report  (MR).  The  first  of  these  two  was  used  only 
for  modifications  entered  into  the  Sundisk  and  Sundance  series 
and  is  illustrated  in  Figure  4-1.  The  other  one  was  used  for 
the  Sundance  and  for  all  later  programs  and  is  illustrated  in 
Figure  4-2. 

Each  MR  documents  one  or  more  software  changes  and  the 
reason  for  the  change(s).  Each  change  resulted  in  a different 
RSM.  In  this  way  a MR  may  have  generated  several  RSMs.  In  some 
cases  a change  went  into  more  than  one  program;  in  that  case  an 
RSM  for  each  was  prepared.  All  modifications  were  also  docu- 
mented in  a memorandum  series  for  each  program.  Frequently 
these  memoranda  were  referenced  by  the  engineer  preparing  the 

basic  data  of  the  RSM,  thus  supplementing  the  information  in 
the  MR. 

A good  deal  of  knowledge,  background  and  judgement  was 

required  and  used  in  interpreting  the  MRs  and  other  documenta- 
tion in  order  to  prepare  the  RSM ' s basic  data,  in  particular  in 
supplying  the  data  for  the  functional  category  field,  the  modi- 
fication category  field  and  the  modification  description  field. 
The  background  knowledge  and  judgement  were  provided  by 

restricting  the  people  preparing  the  data  to  those  with  exten- 
sive experience  in  developing  Apollo  software;  nevertheless,  it 
should  be  recognized  that  a significant  degree  of  subjective 

judgement  was  exercised  in  assigning  categories  to  the  data 
items. 

Each  RSM  contains  9 fields,  which  are  described  below.  In 
all  numeric  fields  leading  zeroes  are  replaced  by  blanks. 
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Figure  4-1  MODIFICATION  REPORT  FOR  SUNDANCE  AND  S'JNDISK 
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MODIFICATION  REPORT  DATE:  1l.l  C ( (.  7 


MODIFICATION  REPORT  FOR  LATER  PROGRAMS 


1 - 4) 


4.1.1  RECORD  IDENTIFIER 


[Columns 


Each  record  identifier  is  unique  in  the  file  of  RSMs.  A 
record  identifier  consists  of  a letter  followed  by  a three  deci- 
mal digit  number.  These  unique  record  identifiers  were  gener- 
ated at  the  time  of  the  preparation  of  the  record.  They  provid- 
ed traceability  during  the  data  preparation  process,  in  that  the 
letter  identifies  the  engineer  who  prepared  the  record.  At  the 
time  of  the  preparation  of  the  RSM , its  record  identifier  was 
written  on  the  MR  from  which  the  RSM  was  prepared,  thus  provid- 
ing references  from  the  file  of  MRs  to  the  file  of  RSMs. 


4.1.2  DATE  OF  MODIFICATION  (Columns  6 - 13) 


An  entry  in  this  field  is  in  the  form  YY-MM-DD,  for  year- 
month-day.  MM  and  DD  may  contain  a leading  blank,  bur  no  lead- 
ing zero.  All  entries  in  this  field  were  checked  to  make  sure 
that  they  were  valid  dates. 

This  date  is  the  date  the  modification  was  submitted' by  the 
programmer  to  the  Assembly  Control  Supervisor  for  approval  and 
inclusion  in  the  program.  The  date  that  the  modification  was 

actually  incorporated  into  the  program  is  not  known,  but  was 
usually  within  a day  or  two,  or  at  most  a week,  of  its  submit- 
tal . 
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In  most  cases  this  date  is  the  date  shown  on  the  MR.  In 
those  few  cases  where  the  date  field  on  the  MR  was  left  blank, 
the  revision  identifiers  (see  paragraph  4.1.3,  below)  were  used 
as  the  basis  for  estimating  this  date.  Estimating  this  date 

sometimes  required  a linear  interpolation,  interpolating  between 
revision  numbers  with  approximately  known  dates. 

4.1.3  REVISION  IDENTIFIER  (Columns  15  - 18) 

This  field  was  taken  directly  from  the  MR  and  denotes  the 
program  revision  into  which  the  modification  was  incorporated. 
The  programs  to  which  the  data  applies  consist  of  six  distinct 
program  series:  Sundisk,  Colossus,  Comanche,  Artemis  (all  Com- 

mand Module  programs),  Sundance  and  Luminary  (Lunar  Module 
programs).  As  shown  in  Table  4-1,  sixteen  separate  flight 

programs  were  manufactured  from  these  series. 

The  revision  identifier  consists  of  a letter  followed  by  a 
rhree  decimal  digit  number.  The  letter  identifies  rhe  program 

series;  the  number  is  rhe  revision  number  within  that  series. 

For  some  revisions  there  are  no  RSMs.  This  could  happen 
for  one  of  two  different  reasons.  (1)  For  Sundisk  revisions  1 
through  88,  and  for  Sundance  revisions  1 through  82  there  were 

no  reports  on  modifications  available.  (2)  In  some  cases  a 


Table  4-1  REVISION  IDENTIFIERS,  FLIGHT  PROGRAMS,  SIZES,  FLIGHTS 


Pi  og . 
Code 

Revi- 

sions 

Program 

Name 

Flight 

Program 

Total 
Wor  ds 

New 

Words 

Space 

Vehicle 

Apollc 

Flight 

D 

1-282 

Sundi  sk 

Sundisk 

36480 

23100 

C 

7 

C 

1-237 

Colossus 

Colossus  1 

37757 

11770 

C 

8 

C 

238-249 

Colossus 

Colossus  1A 

37854 

110 

C 

9 

M 

1-  45 

Comanche 

Colossus  2 

38575 

2035 

C 

10 

M 

46-  55 

Comanche 

Colossus  2A 

38610 

110 

C 

11 

M 

56-  67 

Comanche 

Colossus  2C 

38702 

215  ! 

C 

12 

M 

68-  72 

Comanche 

Colossus  2D 

38702 

34  ! 

C 

13 

M 

73-108 

Comanche 

Colossus  2E 

38402 

1692  ! 

C 

14 

A 

1-  72 

Ar  temi s 

Colossus  3 

38485  * 

770  # 

C 

15 

S 

1-306 

Sundance 

Sundance 

36424 

28600 

L 

9 

L 

1-  69 

Luminary 

Luminary  1 

37904 

10560 

L 

10 

L 

70-  99 

Luminary 

Luminary  1A 

38646 

2310 

L 

11 

L 

100-116 

Luminary 

Luminary  IB 

38502 

700  ! 

L 

12 

L 

117-131 

Luminary 

Luminary  1C 

38502 

150  ! 

L 

13 

L 

132-178 

Luminary 

Luminary  ID 

38202 

940  ! 

L 

14 

L 

179-210 

Luminary 

Luminary  IE 

38452  + 

770  # 

L 

. 
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revision  was  created  without  a MR  being  prepared.  This  happened 
only  in  the  case  of  an  abort  during  the  assembly  process,  where 
due  to  a clerical  error  or  a machine  failure,  a spurious  rev- 
ision was  created.  In  those  cases,  the  assembly  supervisor 
simply  resubmitted  the  original  modifications. 

The  following  revision  numbers  do  not  appear  on  the  RSMs: 

Comanche  45.1,  45.2 
Comanche  72.1,  72.2,  72.3 
Luminary  69.1,  63. 2 
Luminary  99.1 
Luminary  131.1 

The  revision  number  on  the  RSM  contains  only  the  digits  to  the 
left  of  the  decimal  point;  thus  a modification  to  revision 
Luminary  69.1,  for  example,  is  recorded  in  the  RSM  as  occurring 
in  L 69.  Fewer  than  15  RSMs  contain  these  truncated  revision 

numbers.  These  revisions  are  the  result  of  re-releases  that 
were  made  after  further  development  had  continued  on  the  prog- 
ram. To  illustrate:  Luminary  69  was  released  for  manufacture; 

meanwhile  Luminary  70,  71,  72,  etc.  were  being  created  (as 
updates  of  Luminary  69).  Testing  of  the  released  program  con- 
ducted during  this  period  revealed  a problem  serious  enough  to 
warrant  re-release.  Apollo  support  software  allowed  the 
retrieval  of  any  previous  revision;  revision  69  was  therefore 
retrieved,  copied,  and  corrected  without  disturbing  the  later 
revisions.  The  offshoot  revision  was  entitled  Luminary  69.1, 

which  later  spawned  Luminary  69.2  when  still  another  serious 
problem  surfaced. 

4.1.4  REFERENCE  (Columns  20  - 25) 

For  about  13%  of  the  RSMs,  another  document  is  referenced. 
The  document  is  identified  in  the  reference  field  in  the  manner 
shown  in  Table  4-2.  The  referenced  document  is  either  a Prog- 
ram Change  Request,  a Program  Change  Notice,  an  Anomaly  Report 
or  an  Assembly  Control  Board  Request  (see  paragraph  3.4).  The 
document  established  the  basis  for  the  change.  If  it  was  writ- 
ten before  the  MR,  then  it  served  to  initiate  the  change.  If 
after  the  MR,  then  it  served  to  rationalize  the  change. 

When  such  a document  was  cited  in  an  MR,  it  was  recorded 
in  the  reference  field  of  the  corresponding  RSM. 

For  the  remaining  RSMs  the  reference  field  is  blank. 
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Table  4-2  REFERENCE  NUMBERS 


Columns 

20-25 

Meaning 

Coun  t 

PCRddd 

Program 

Change  Request 

ddd 

949 

PReddd 

Program 

Change  Request 

eddd 

II 

PCNddd 

Program 

Change  Notice 

ddd 

78 

PNeddd 

Program 

Change  Notice 

eddd 

II 

COLddd 

Colossus 

Anomaly 

ddd 

40 

ACBxxx 

Assembly 

Control  Board 
Request 

XXX 

357 

COM  d d d 

Comanche 

Anomaly 

ddd 

43 

LNYddd 

Luminary 

Anomaly 

ddd 

40 

symbol  used  explanation 

above 


d 

e 

x 


a decimal  digit 

a decimal  digit  other  than  zero 
a decimal  digit  or  a capital  letter 


.1.5  FUNCTIONAL  CATEGORY  (Column  27) 

This  field  identifies  the  function  within  the  flight  prog- 
am which  is  being  modified.  22  functional  categories  were 

dentified  and  assigned  code  letters  "A"  through  "V".  The  func- 
ional  categories  and  their  codes  are  shown  in  Table  4-3.  The 
ield,  then»  contains  a letter  denoting  the  functional  category 
f the  code  being  modified. 

This  field  was  coded  on  the  basis  of  the  information  in 
he  MR/  supplemented  by  other  documentation  and  the  background 
nowledge  of  the  individual  preparing  the  data. 

.1.6  MODIFICATION  CATEGORY  (Columns  29  - 32) 

The  entry  in  this  field  denotes  that  category  out  of  the 
08  preselected  categories  (shown  in  Table  4-4)  which  best 

escribes  the  modification  or  the  reason  for  the  modification, 
he  entry  itself  is  a code  denoting  the  modification  category 
nd  consists  of  a letter  followed  by  a three  decimal  digit 
umber . 

184  categories  were  defined  by  the  sponsor  in  the  Statement 
f Work/  each  with  its  unique  code.  First  these  codesf  which 
ere  5 characters  long/  with  the  first  and  second  character 

lways  identical/  were  changed  by  deleting  the  second  redundant 
haracter  of  the  code  to  shorten  it/  and  by  replacing  leading 
eroes  in  che  numerical  portion  of  the  code  by  blanks. 

Secondly/  u became  apparent  in  che  course  of  this  scudy 
hat  a large  proportion  of  Apollo  programming  errors  do  not  fit 
he  184  categories  defined  by  the  sponsor.  This  mismatch  is 

ue  primarily  to  the  fact  that  the  Apollo  code  was  written  for 
real-time  mul tipr ogr ammed  application/  whereas  the  error  data 
nalyzed  previously  under  the  sponsor's  auspices  was  derived 
torn  non  real-time  applications.  Real-time  errors  manifest 
hemselves  as  r outine- to-routine  interface  conflicts  as  well  as 
riority  or  time  conflicts.  The  sequence  of  operation  of 
outines  in  real  time  is  frequently  dependent  upon  the  proper 
etting  of  flags  and  other  interface  data  by  other  routines, 
rogram  sequencing  and  restart  protection  is  considerably  com- 
licated  by  the  actions  of  an  on-line  astronaut.  Finally,  the 
xistence  of  an  interactive  user  creates  a series  of  possible 
an/machine  interface  problems. 

In  order  to  avoid  forcing  the  Apollo  real-time  errors  into 
nnatural  categories,  24  new  categories  were  defined,  each  with 
new  code*  5 other  categories  had  their  descriptions  modified 
ut  not  their  code;  one  category  had  its  description  modified 
nd  was  given  a new  code;  and  1 category  was  given  a new  code 
ithout  its  description  being  modified.  We  thus  arrive  at  the 
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Table  4-3  FUNCTIONAL  CATEGORIES,  THEIR  CODES  AND 


Colossus  Luminary 

Code  Functional  Category  237  98 


\ Digital  Autopilot 


5815 


3464 


B Error  Detection/Recovery 


C Crew  Interface 


2220 


D Telemetry 
E I/O 


F Executive 


1173 


G Sequence  Initialization/Reinitialization  1009 


H Display 


3603 


3107 


I Naviga  tion 


4043 


4655 


J Coordinate  Transforms 


K Vehicle  Attitute  Computations/Maneuvers 


L Tracking 


2929 


3560 


M Targeting 


3728 


2684 


N Powered  Flight  Maneuvers 


2479 


3103 


0 Guidance  Computations 


1570 


P lAlignments 


i 588 


2101 


D Interpreter 


2145 


2150 


R iMath  Subroutines 


3 lSoftware  Utility  Routines 


r Hardware  Failure  Monitor 


1780 


U Hardware  Service  Routines 


1982 


V Manual  Operations  (non-software) 


"NA"  means  that  the  size  of  the  code  for  the  functional 
category  is  not  available. 


Table  4-4  MODIFICATION  CATEGORIES 
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final  count  of  208  categories.  Care  was  taken  to  avoid  prolif- 
eration of  categories.  In  most  cases,  the  new  real-time  cate- 
gories are  simply  subsets  of  existing,  closely  related  non  real- 
time categories.  Descriptions  of  categories  were  modified  to 
clarify  their  meaning  within  the  Apollo  context.  Categories 
were  recoded  to  provide  a clearer  hierarchy  of  categories.  All 
these  changes  are  shown  in  Table  4-5. 

The  value  of  the  modification  category  field  was  entered 
by  the  RSM ' s preparer  on  the  basis  of  1)  all  the  information  in 
the  MR,  2)  other  documentation  related  to  the  modification,  3) 
using  his  or  her  knowledge  of  the  context  in  which  the  MR  had 
been  written  and  4)  using  his  or  her  knowledge  of  the  categor- 
ization scheme  that  was  being  used. 

The  MRs  document  each  change  to  an  assembly  and  the  reason 
for  the  change.  Ideally  it  would  be  possible  for  experienced 
Apollo  programmers  to  go  through  the  MRs  categorizing  each  modi- 
fication according  to  the  categorization  scheme  being  employed. 
However,  when  the  hand-written  MRs  were  originally  produced, 
they  were  not  produced  with  anything  like  the  present  purposes 
(e.g,  carrying  out  error  analysis)  in  mind.  As  it  is,  the  MRs 
frequently  do  not  contain  sufficient  information  to  categorize 
a modification  and  thus  other  documentation  is  required.  The 
appropriate  Apollo  memo  series  was  used  for  this  purpose,  since 
that  series  contained  memos  provided  by  the  Assembly  Control 
Supervisor  which  gave  for  each  revision  a detailed  description 
of  all  modifications  incorporated  in  the  revision.  Also,  if  the 
MR  references  a PCR,  PCN , Anomaly  Report  or  an  Assembly  Control 
Board  Request  (see  paragraph  4.1.4),  supporting  data  identify- 
ing the  source  of  the  change  was  obtained,  if  needed,  from  the 
referenced  document.  Even  with  all  this  documentation,  often 
not  enough  information  was  available  about  the  nature  of  the 
original  change  and  about  the  context  in  which  the  change  was 
made?  in  many  cases,  judgement  based  on  experience  was  relied 
on  • 

4.1.7  MODIFICATION  DESCRIPTION  (Columns  34  - 83) 

This  field  contains  a brief  textual  description  of  the 
program  change  or  the  reason  for  the  change.  Generally  this  is 
just  a fuller  description  of  the  modification  or  the  reason  for 
it  then  what  is  denoted  in  the  modification  category  field 
(paragraph  4.1.6).  (Nore  that  "reason  for  the  modification" 
and  "description  of  the  modif ica tion"  are  largely  synonymous.) 

The  description  in  this  field,  like  the  code  in  the  modi- 
fication category  field,  was  entered  by  the  RSM ' s preparer  on 
the  basis  of  all  the  information  in  the  MR,  other  available  doc- 
umentation, as  well  as  his  or  her  knowledge  of  the  context  in 
which  the  MR  had  been  written. 
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Table  4-5  CHANGES  TO  THE  ERROR  MODIFICATION  CATEGORIES 


Code  I Sta tus  f Ca tegorv  Description.  (Explanation  of  the  Change) 


A 42 


A130 


B 63 


Sampled  data  problem;  characteristic  of  real-time 
data  was  not  as  expected. 

Fixed  point  scaling  error. 

Open  branch;  possibility  of  logic  branch  to 
undefined  location. 


B 70 


Incorrect  log;.c  design. 

("design"  added  to  differentiate  category  from 
the  one  below  coded  "B  71") 


B 71 


D 20 


Incorrect  implementation;  logic  designed 

correctly  but  incorrectly  implemented. 

Data  written  in  or  read  from  wrong  memory 
loca  tion* 

("disk"  changed  to  "memory") 

Program  segmentation  (layout  of  subroutines,  etc.) 

(Original  category  was  specialized  to  apply 
only  to  changes  to  the  organization  of 
subroutines) . 

Real  time  organization  problem  (incompatible 

modes);  incompatible  modes  of  operation  of 
routines  in  real  time  (includes  restart 
problems)  • 

Organization  errors  (location  of  code,  etc.); 

errors  in  location  of  code,  deletion  of 
unused  code,  etc- 

Real  time  routine/routine  initialization  error; 
initialization  error  resulting  in 
real-time  sequencing  problem. 

Priority  conflict;  error  in  establishment  of 
priority  of  real-time  routine. 

Time  conflict;  real-time  routine  does  not  meet 
time  constraints. 


Inadequate  restart  capability  (was  J100);  addition 
or  modification  of  restart  tables,  etc. 


* 


Table  4-5  CHANGES  TO  THE  ERROR  MODIFICATION  CATEGORIES  (cont.) 


Code 

Status 

Category  Description.  (Explanation  for  the  Change) 

H 50 

A 

Errors  in  restart  logic;  modification  in  restart 
procedures. 

J 10 

c 

Incompatibility  with  external  requests;  changes 
made  to  conform  to  external  requests. 

J 90 

C 

Poor  design  in  man/machine  interface, 
("operator"  changed  to  "man/machine") 

J100 

D 

Inadequate  interrupt  and  restart  capability. 
(Code  for  the  category  changed  to  "H  40") 

K 11 

D 

Uncoordinated  use  of  data  elements  by  more  than 
one  user. 

(Code  for  the  category  changed  to  "K  20") 

K 20 

A 

Uncoordinated  use  of  data  elements  by  more  than 
one  user. 

(Code  for  this  category  was  "K  11"  originally) 

L 26 

A 

Man/machine  real-time  spec  change;  user  requested 
change  in  man/machine  interface. 

L 27 

A 

Error  recovery  spec  change;  user  requested  change 
in  restart  mechanism. 

L 90 

A 

Implementation  of  original  spec.;  new  code  required 
to  conform  to  original  specification. 

N 60 

A 

Make  room  in  E-Bank  (erasable  memory). 

N 70 

A 

Reorganize  data. 

N 80 

A 

Missing  data  definition;  primarily  includes  those 
modifications  to  the  Erasable  Assignments 
that  parallel  coding  changes  that  fall 
into  such  categories  as  "incorrect 
implementation  of  logic  design",  "missing 
logic",  and  others  of  a similar  nature. 

S 10 

A 

Requirements  error  (insufficient,  inadequate); 

coding  changes  due  to  an  insufficient  or 
inadenuate  specification. 

S 20 

A 

Requirements  enhancement;  coding  changes  due  to 
changes  in  requirements. 

J 
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\ Table  4-5  CHANGES  TO  THE  ERROR  MODIFICATION  CATEGORIES  (cont.) 


Code 

Status 

Category  Description;  Explanation  for  the  Change 

V 0 

A 

IN-HOUSE  PROGRAM  IMPROVEMENTS. 

V 10 

A 

Simplified  interface  and/or  convenience. 

(Description  same  as  for  the  category  coded 
"L  10") 

V 20 

A 

New  and/or  enhanced  functions. 

(Description  same  as  for  the  category  coded 
"L  20") 

V 30 

A 

Data  base  management:  and  integrity. 

(Description  same  as  for  the  category  coded 
"L  70") 

V 40 

A 

CPU  time  optimization. 

V 50 

A 

Program  memory  optimization. 

Legend  for  column  headed  "Status": 


"A"  means  that  the  code  was  added. 

"C"  means  that  the  description  was  changed, 

but  the  code  was  not. 
"D"  means  that  the  code  was  deleted. 


40 


4.1.8  FLIGHT  PROGRAM  {Columns  85  - 87) 

A modification  is  intended  for  a particular  flight  program 
then  under  development.  This  study  covers  the  development  of  16 
different  flight  programs. 

The  field  has  one  of  16  values,  and  indicates  that  the  mod- 
ification is  intended  for  the  flight  program  denoted  by  the 
value  and  under  development  at  the  time  of  the  modification. 
Table  4-1,  page  30,  shows  which  value  denotes  which  flight 
program- 

This  field  also  indicates  that  the  modification  was  intend- 
ed for  a particular  space  vehicle  (paragraph  4. 1.8.1)  and  a 
particular  Apollo  flight  (paragraph  4. 1.8. 2).  However,  there 
are  no  modifications  that  were  intended  for  the  Lunar  Module  of 
Apollo  flight  7 or  8,  because  there  was  no  Lunar  Module  on 
Apollo  flights  7 and  8.  Thus,  instead  of  18  (i-e-,  2*9)  possi- 

ble combinations  of  vehicle  and  flight  number,  there  were  only 
16. 


The  entry  for  this  field  is  computed  from  the  revision 
identifier  (see  paragraph  4.1.3),  using  the  information  in 
Table  4-1. 

4. 1.8.1  SPACE  VEHICLE  (Column  85) 

This  field  has  one  of  two  values.  "C"  denotes  that  the 
modification  was  made  to  a Command  Module  program,  "L",  to  a 
Lunar  Module  program. 

The  entry  for  the  field  was  derived  from  the  leading 
character  of  the  revision  identifier  (see  paragraph  4-1.3), 
using  the  information  in  Table  4-1. 

4. 1.8. 2 FLIGHT  NUMBER  (Columns  86  - 87) 

A flight  program  (rope)  is  designed  to  play  its  part  in 
one  or  more  Apollo  flights.  The  Apollo  flights  are  numbered. 

The  flight  programs  covered  in  this  study  flew  on  Apollo  flights 
7 through  17.  Note  that  the  two  flight  programs  intended  for 

Apollo  flight  15  are  also  for  Apollo  flights  16  and  17,  since 
these  three  flights  employed  the  same  programs. 

This  field  identifies  for  which  one  of  these  Apollo  flights 
the  modification  was  first  implemented,  by  containing  in  the 
field  a two  decimal  digit  number  identifying  the  Apollo  flight. 

The  entry  for  this  field  was  derived  from  the  revision 
identifier  (see  paragraph  4.1.3),  using  the  information  in 
Table  4-1. 
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4.1.9  SOFTWARE  DEVELOPMENT  PHASE  (Column  89) 

This  field  indicates  whether  the  modification  was  made 
during  the  development  or  during  the  verification  phase  of  the 
flight  program's  software  development  cycle-  The  letter  "D" 
entered  in  the  field  indicates  that  the  change  was  made  during 
the  development  phase,  the  letter  "V",  during  the  verification 

phase . 

Note  that,  if  the  modification  is  to  correct  an  error,  then 
this  field  does  not  generally  indicate  (neither  does  any  other 
field)  in  which  phase  of  the  software  development  cycle  the 

error  was  found  (recognized).  Such  data  is  not  generally  avail- 
able. However,  it  is  safe  to  assume  that  in  most  cases  the 

error  was  discovered  in  the  same  phase  that  it  was  corrected. 
This  is,  of  - course,  necessarily  the  case  if  the  error  was 

corrected  during  the  development  phase. 

Note  further  that,  if  the  modification  is  to  correct  an 
error,  then  this  field  does  not  generally  indicate  in  which 

phase  of  the  software  development  cycle  the  error  was  intro- 
duced into  the  program.  However,  the  modification  category 

field  generally  indicates  whether  an  error  was  introduced 
during  the  specification  phase  or  later. 

The  value  of  this  field  was  derived  from  two  other  values 
contained  in  the  RSM , (1)  the  date  the  modification  was  complet- 

ed (paragraph  4.1.2),  and  (2)  the  flight  program  (paragraph 
4.1.8),  as  well  as  from  (3)  the  date  the  verification  phase 

began  for  that  flight  program-  The  value  of  the  field  is  "D" 
if  the  date  the  modification  was  submitted  was  earlier  than  the 
date  of  the  beginning  of  the  verification  phase;  otherwise  the 
value  of  the  field  is  "V". 

The  date  the  verification  phase  began  is  the  date  that  is 
underlined  in  Table  4-6  for  that  flight  program.  This  date  is 
either  the  date  that  configuration  control  by  NASA  began,  if 

that  date  is  available,  or  else  the  date  that  Level  4 testing 
started-  The  NASA  configuration  control  date  was  preferred,  if 
available,  over  the  Level  4 testing  date,  because  it  is  consid- 
ered a more  accurate  and  reliable  estimator  of  the  beginning  of 
the  verification  phase. 

Other  and  more  refined  breakdowns  of  that  portion  of  the 
software  development  cycle  during  which  program  modifications 
were  made  were  considered,  but  were  rejected,  either  because 
the  resulting  phases  would  not  be  sufficiently  meaningful  or 
because  the  available  data,  whether  on  the  basis  of  partition- 
ing the  series  of  revisions  into  phases  or  on  the  basis  of 
establishing  dates  that  separate  phases,  was  not  sufficiently 
reliable • 
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Table  4-6  FLIGHT  PROGRAM  RELEASE  DATES 


Release 
Da  te 


FlighL 

Space 

Program 

Vehicle 

Sundi sk 

Command 

Module 

Colossus 

Command 

1 

Module 

Colossus 

1A 

Colossus 

2 

Colossus 

2 A 

Colossus 

2C 

Colossus 

2D 

Colossus 

2E 

Colossus 

3 

Sundance 

Luminary 

1 

Luminary 

1A 

Luminary 

IB 

Luminary 

1C 

Luminary 

ID 

Luminary 

IE 


Command 
Module 
Command 
Module 
Command 
Module 
Command 
Module 
Command 
Modul  e 
Command 
Module 
Command 
Module 
Lunar 
Modul  e 
Lunar 
Module 
Lunar 
Module 
Lunar 
Module 
Lunar 
Module 
Lunar 
Module 
Lunar 
Module 


67- 10-  4 

68-  7-26 
68-  8-23 


NA 

NA 

NA 

NA 

NA 

68-  4-12 


NA 


68-  2 
68-  8 


67-10-28 


- 2 
-18 
-18 


69-12-12 


70-  6-29 

71-  3-  1 
68-10 
69-  4-  2 
69-  6-17 

69-  8-12 

70-  2-  6 

70-  4-18 

71-  3-20 


Star  t 

Complete 

NC 

NC 

NC 

NC 

NC 

NC 

NC 

NC 

69-  3-14 

69-  4-  4 

69-  7-  7 

69-  7-17 

69-10-10 

69-10-24 

70-  5-  8 

70-  5-26 

70-  7-20 

71-  1-28 

NC 

NC 

68-11-2 2 

68-11-22 

69-  2-28 

69-  4-14 

69-  7-14 

69-  8-12 

69-10-20 

69-11-  5 

70-  4-13 

70-  5-24 

70-12-  7 

71-  1-18 

Legend  for  date  fields: 


NA  Date  not  available 

NC  Date  not  compiled,  but  probably  available 


4-2  SUPPLEMENTAL  DATA 


The  data  described  in  this  section  was  derived  from  CSDL 
Apollo  project  documentation-  It  is  contained  in  the  second 

file  delivered  to  the  sponsor.  This  file  contains  83  records 

of  80  characters  each.  The  order  of  the  items  in  each  record 


is  listed 

below: 

Field 

1 

Columns 

1-5 

paragraph  of  the  sponsor's 
Statement  of  Work 

Field 

2 

Columns 

7-16 

alphanumeric  tag 

Field 

3 

Columns 

18  - 29 

numeric  value 

or 

Field 

3 

Columns 

18  - 19 

NA  (for  not  available) 

or 

Field 

3 

Columns 

18  - 76 

multiple  subfields  (computer  hours 
table) 

Field 

4 

Columns 

31  - 80 

comment  (for  records  other  than  those 
containing  computer  hours  table) 

The  information  contained  in  this  file  is  described  below. 

4.2.1  ERRORS 

No  errors  were  due  to  failures  in  the  computer  hardware 

only. 


The  number  of  software  errors  that  resulted  in  abnormal 
processor  termination  is  not  available,  since  such  data  would 
have  to  be  based  on  a detailed  history  of  digital  simulation 
runs. 


The  number  of  software  errors  that  resulted  in  normal 
processor  termination  is  not  available,  since  such  data  would 
have  to  be  based  on  a detailed  history  of  digital  simulation 
runs. 


There  were  no  software  errors  for  which  the  exact  cause  of 
the  error  was  unknown  when  the  corresponding  software  problem 
report  was  closed,  since  such  reports  were  not  closed  until  the 
problem  was  understood. 

4.2.2  MACHINE  USE  STATISTICS 

The  total  amount  of  CPU  time  used  for  the  Apollo  project 
per  month  is  shown  in  Figures  4-3  and  4-4  and  Table  4-7.  This 
CPU  time  is  provided  only  for  Draper  Laboratory's  Honeywell 
1800  and  its  IBM  360/75,  and  not  for  the  Draper  Laboratory's 
hybrid  computer,  its  System  Test  Laboratory,  or  testing  facili- 
ties used  outside  the  Laboratory.  The  CPU  time  given,  for  the 
Honeywell  1800  and  the  IBM  360/75,  is  in  IBM  360/75  equivalent 
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hours.  (4  hours  of  Honeywell  1800  time  are  considered  equiva- 
lent to  1 hour  of  IBM  360/75  time.)  The  total  amount  of  CPU 
time  used  for  the  Apollo  project  is  provided  on  a monthly 
basis,  not  on  a daily  basis- 

The  monthly  CPU  time  is  based  on  Apollo  project  documenta- 
tion . 


Note  that  the  CPU  time  used  for  the  Apollo  project  went 
mainly  into  (1)  testing  (i.e.,  finding  errors),  (2)  assembling 
programs  to  correct  errors,  (3)  assembling  programs  to  modify 
the  programs  to  meet  new  requirements  and  (4)  developing  the 
equations.  Thus,  except  for  the  time  that  went  into  developing 
the  equations — a rather  small  proportion  of  the  total  time  used, 
the  preponderance  of  the  CPU  time  used  for  the  Apollo  project 
went  into  finding  and  correcting  errors  or  effecting 
requirements  changes- 

4.2.3  NUMBER  OF  SIMULATION  RUNS 

Data  on  the  number  of  simulation  runs  for  each  period  in 
the  software  development  cycle  is  not  available;  the  record  of 
this  item,  therefore,  contains  the  letters  "NA". 


Even  if  data  regarding  the  number  of  simulation  runs  were 
available,  it  is  not  clear  how  meaningful  this  data  would  be, 
since  the  simulation  testing  done  at  Draper  on  Draper's  IBM 
360/75  and  on  Draper's  Honeywell  1800  represented  only  a small 
portion  of  all  the  simulation  testing  that  was  done.  A great 
deal  of  simulation  testing  was  also  done  at  Cape  Kennedy, 
NASA/JSC,  Grumman,  Rockwell,  Delco  and  on  Draper's  hybrid  comp- 
uter and  in  Draper's  System  Test  Laboratory.  (These  remarks 

also  apply  to  the  data  discussed  under  paragraph  4-2.2,  above.) 


4.2.4  SPEED  OF  SIMULATION 


The  simulator  to  real  time  processing  ratio  varied 
considerably,  depending  on  the  rate  of  activity  of  the  AGC  being 
simulated.  At  best,  when  the  AGC  (the  on-board  computer)  was 
performing  the  coasting  flight  functions  (i-e.,  when  only  a 
small  percentage  of  its  processing  capacity  was  being  used), 
about  1 unit  of  IBM  360/75  rime  was  required  to  simulate  8 units 
of  AGC  rime.  At  worst,  when  the  AGC  was  very  heavily  loaded 
(i-e.,  all  of  i rs  processing  capability  was  being  used),  about 
4 unirs  of  IBM  360/75  rime  was  required  ro  simulate  1 unit  of 
AGC  rime-  With  an  average  load  on  rhe  AGC,  about  3 unirs  of 

IBM  360/75  rime  were  required  ro  simulate  2 unirs  of  AGC  time. 


These  figures  remained  fairly  constant  over  the  period 
covered  by  this  study.  The  Honeywell  1800  program  that  simu- 
lated the  AGC  was,  when  correcting  for  the  4 to  1 ratio  in 
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processing  power  between  the  IBM  360/75  and  the  Honeywell  1800, 
approximately  as  efficient  in  simulating  the  AGC  as  the  IBM 
360/75  simulator  program. 

The  user  of  the  simulator  program  was  able  to  vary  the 
fidelity  (high  and  low)  with  which  the  environment  was  simulat- 
ed. Users  specified  high  fidelity  only  when  high  accuracy  was 
required,  thus  effecting  some  savings  in  run  time. 

4.2.5  SOFTWARE  CHARACTERISTICS 

4. 2. 5. 1 SIZE 

The  total  number  of  computer  words  in  all  the  16  releases 
covered  by  this  study  is  approximately  610,000. 

A second  estimate  of  the  total  size  of  the  Apollo  flight 
software  covered  by  this  study  is  provided  by  the  sum  for  all 
ropes  of  the  number  of  words  added  or  changed  since  the  last 
rope,  excluding,  however,  all  words  changed  to  correct 
programming  errors.  This  number  is  83,866. 

The  second  of  these  two  numbers  is  a more  meaningful  esti- 
mate of  the  total  size  of  the  Apollo  project,  if  we  consider 
primarily  the  extent  of  the  development  effort.  However,  the 

first  is  the  more  meaningful  estimate,  if  we  consider  the  size 
of  the  testing  effort,  since  every  rope  had  to  be  tested  anew, 
and  there  was  relatively  little  carry  over  to  the  testing  of 
one  rope  from  the  testing  of  ics  parent.  In  particular,  approx- 
imately the  same  amount  of  Level  4 through  Level  6 testing  was 
performed  for  each  rope- 

The  first  of  these  two  numbers  can  be  fairly  well 
approximated  by  multiplying  the  size  of  the  computer's  memory 
(38,912  words)  by  the  number  of  ropes  (16,  i.e.,  one  rope  for 

each  of  the  Apollo  flights  7 and  8 and  two  ropes  for  each  of 
the  flights  9 through  15).  This  provides  an  estimate,  albeit 
high,  of  622,592  words.  However,  a more  accurate  estimate, 
610,000  words,  was  obtained  by  using  the  data  of  reference  11. 
The  sizes  of  Colossus  3 and  Luminary  IE  are  not  available  in 
reference  11.  The  total  number  of  words  for  flight  program 
Colossus  3 was  taken  from  the  program  listing  for  Artemis  revi- 
sion 72.  The  total  for  Luminary  IE  was  estimated  by  an  exper- 
ienced Assembly  Control  Supervisor.  Table  4-1,  page  30,  lists 
the  sizes  for  each  rope. 

The  best  estimate  of  the  second  of  these  numbers  is  the 
sum  of  the  entries  of  the  sixth  column  (headed  "New  Words”)  of 
Table  4-1.  This  column  lists,  for  each  of  the  16  flight  prog- 
rams, an  estimate  of  the  number  of  words  added  or  changed  since 
the  last  flight  program  (rope)  for  the  same  space  vehicle, 
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excluding,  however,  all  words  changed  to  correct  programming 
errors.  8 of  the  16  numbers  of  this  column  are  taken  from 
reference  12.  6 of  the  16  are  taken  from  reference  7.  The 

other  2 were  estimated  by  averaging  the  numbers  in  the  column 
for  the  previous  4 flights. 

For  an  explanation  of  why  size  is  estimated  in  terms  of 
numbers  of  computer  words,  see  the  discussion  in  paragraph 

4. 2. 5. 1. 1. 

4. 2. 5. 1.1  SIZE  OF  FUNCTIONAL  CATEGORIES 

Table  4-3,  page  34,  provides  an  estimate  of  the  amount  of 
code  (in  numbers  of  words)  used  for  each  functional  category  for 
two  representative  revisions,  Colossus  revision  237  (the  final 
revision  for  the  Command  Module  for  Apollo  flight  8),  and  Lumin- 
ary revision  98  (one  of  the  later  revisions  for  the  Lunar  Module 
for  Apollo  flight  11).  Where,  for  a size,  "NA"  is  entered 
instead  of  a number  it  means  that  the  size  of  the  code  for  the 
functional  category  is  not  available,  since  the  code  is  embedded 
among  the  code  that  was  counted  for  other  functional  categories. 

These  numbers  have  not  been  computed  for  the  other  ropes, 
because  the  program  listings  on  the  basis  of  which  these  numb- 
ers are  computed  are  not  readily  available.  The  NASA  archives 
in  Houston  may  contain  a complete  or  near  complete  set,  but 
this  has  not  been  investigated. 

The  size  of  each  component  module  (i.e.,  the  amount  of 
code  used  for  each  functional  category)  is  stated  in  terms  of 
number  of  machine  words  (each  word  consisting  of  15  bits  of 
information,  with  a sixteenth  bit  being  used  as  a parity  check), 
and  includes  constants  (data)  and  interpretive  code,  as  well  as 
instructions.  Although  a listing  of  a program  indicates  for 

each  word  whether  the  word  is  an  instruction  word  (the  word  is 
unmarked),  a constant  (the  word  is  marked  with  a "C"),  or  a word 
of  interpretive  code  (the  word  is  marked  with  an  "I"),  it  would 
still  be  an  arduous  task  to  count  these  separately  (for  each 
rope  over  30,000  words  would  have  to  be  categorized),  and  no 
clear  benefit  would  be  obtained  from  this.  Typically  and  very 
roughly,  about  53%  of  the  words  of  a program  represent  basic 
machine  instructions,  34%  represent  interpretive  code  and  13% 
constants. 

Measuring  size  in  terms  of  machine  words  conforms  to  the 
sponsor's  requirement  that  programs  be  measured  in  terms  of 
number  of  machine  instructions  for  that  portion  of  a program 
written  in  assembly  language-  The  AGC  instructions  can  be 
meaningfully  viewed  as  being  of  fixed  (i.e*,  constant)  length, 
each  the  size  of  a word.  Thus  it  makes  sense  to  measure  the 
number  of  machine  instructions  in  terms  of  the  number  of  words 


50 


the  machine  instructions  occupy.  (This  makes  sense  even  in 
terms  of  those  machine  instructions  that  employ  more  than  one 
word.  For  example)  a conditional  branch  instruction  requires  4 
words/  yet  those  4 words  are  generally  occupied  by  4 
instruc  tions. ) 

Measuring  size  in  terms  of  machine  words  conforms  to  the 
sponsor's  requirement  that  programs  be  measured  in  terms  of 
number  of  lines  of  source  code  for  that  portion  of  a program 
written  in  a higher  order  language/  since  each  line  of  the  list- 
ing representing  interpretive  code  can  be  considered  to  be  a 
line  of  source  code  and  corresponds  to  a machine  word. 

» 

Further/  since  interpretive  code,  machine  instructions  and 
constants  are  apt  to  be  interspersed  throughout  a log  section/ 
subroutine  or  functional  category/  it  makes  sense  to  use  the 
same  yardstick  for  interpretive  code  as  for  machine  instruct- 
ions. Note  that  this  situation  is  different  from  the  typical 

onef  where  different  modules  are  written  in  different  languages/ 
in  that  one  segment  of  code  is  often  written  partly  in  a higher 
order  language  (interpretive  code)  and  partly  in  a (more 
normal)  assembly  language. 

Since  modules  are  viewed  as  functional  categories , the 
size  of  each  module  given  here  is  based  on  an  approximation 
obtained  by  considering  each  "log  section"*  as  containing  one 
function.  To  the  extent  that  this  is  true/  the  sizes  are  cor- 
rect. It  should  be  recognized/  however/  that/  although  log 

sections  in  general  were  assigned  on  a functional  basis,  cert- 
ain code  embedded  in  any  given  log  section  properly  belonged  to 
a function  other  than  that  represented  by  the  log  section.  As 
an  example,  a log  section  that  was,  quite  reasonably,  judged  to 
belong  to  the  functional  category  of  "navigation"  (since  a very 
large  proportion  of  its  code  is  dedicated  to  that  function)  , 
contains  within  it  some  code  chat  is  ''display",  some  that  is 
"I/O",  and  perhaps  some  other  categories  as  well.  Since  the 
functional  category  assigned  to  the  code  modification  (column  27 
in  the  first  data  file)  was  assigned  on  the  basis  of  actual 
function,  there  is  an  inconsistency  between  the  sizes  given 
here  and  the  modification  functional  categories. 

The  first  step  in  computing  the  size  of  the  functional 
categories  for  the  two  programs  was  to  assign  each  log  section 
of  each  of  the  two  programs  to  one  (or,  in  some  cases,  to  none 
or  to  two)  functional  categories,  depending  on  which  category 
(or  categories)  best  described  the  log  section's  principal 
function.  The  result  of  this  step  is  shown  in  the  Table  4-8 


*An  explanation  of  "log  section"  and  " subroutine"-- these  fields 
appear  on  the  Modification  Report  (MR)--is  given  in  Appendix  B. 
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headed  "Functional  Categories  Applied  to  Colossus  237"  and  Table 
4-9  headed  "Functional  Categories  Applied  to  Luminary  98".  The 
second  step  was  to  take  the  listing  for  each  of  the  two  programs 
in  turn#  look  up  on  the  listing  the  size  of  each  log  section 

and  add  this  number  to  the  size  for  the  appropriate  functional 
ca  tegor y • 

One  minor  problem  connected  with  this  procedure  arises 
from  the  fact  that#  for  each  of  the  programs,  just  2 (out  of 

some  80)  log  sections  are  assigned  to  more  than  one  functional 
category.  In  particular,  in  both  Colossus  revision  237  and  in 
Luminary  revision  98  the  log  section  T4RUPT  PROGRAM  is  assigned 
to  functional  category  "E"  (I/O)  as  well  as  "T"  (Hardware  Fail- 
ure Monitor),  and  IMU  COMPENSATION  PACKAGE  assigned  to  "E"  and 

"U"  (Hardware  Service  Routine).  Clearly  we  do  not  want  to  add 

the  size  of  a log  section  to  the  size  of  each  of  two  functional 
categories.  There  are  at  least  the  following  3 ways  of 
resolving  this  problem:  (1)  We  arbitrarily  split  the  size  of 

the  log  section  in  two,  and  add  half  to  one  functional  category 
and  half  to  the  other-  (2)  We  could  specify,  for  each  log 
section  which  is  assigned  to  more  than  one  functional  category, 
what  proportion  it  is  most  appropriate  to  assign  to  one  and  what 

to  the  other.  (3)  We  assign  the  size  of  the  log  sections  to  the 

more  appropriate  of  the  two  categories.  We  chose  the  third  of 
these  3 ways*  The  reasoning  was  as  follows:  All  the  log  sect- 

ions in  question  involved  I/O.  Input/Output  is  a more  subsidi- 
ary function  than  is  Hardware  Services  and  Hardware  Failure  Moni- 
tor , and  is  thus  best  subsumed  under  (embedded  in)  the  more 
primary  function.  Input/Output  may  not  be  a meaningful  func- 
tional category  in  the  first  place. 

Tables  4-8  and  4-9  were  prepared  by  two  engineers  with 
extensive  experience  with  Apollo  software.  For  each  log  section 
the  functional  category  (see  paragraph  4.1*5)  was  selected  which 
best  described  the  function  to  which  the  log  section  contribut- 
ed. Some  log  sections  consisted  only  of  constants,  which  parti- 
cipated in  many,  if  not  all  functions.  The  functional  category 
column  in  the  table  for  these  log  sections  is  marked  "all".  The 
log  section  ENTRY  LEXICON  of  Colossus  237  consisted  of  a comment 
and  did  not  produce  executable  code,  hence  could  not  be  assigned 
to  a functional  category,  and  is  marked  "comment". 

4. 2. 5. 2 MODE  OF  CONSTRUCTION 

All  the  16  flight  programs  were  developed  using  convention- 
al programming  techniaues-  Structured  programming  was  not  in 
general  use  at  the  time  of  the  Apollo  development- 
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Table  4-8  FUNCTIONAL  CATEGORIES  APPLIED  TO  COLOSSUS  2 37 


Log  Section 

Functional 
Category 
( coded ) 

KILERASE 

ERASABLE  ASSIGNMENTS 

all 

INTERRUPT  LEAD  INS 

F 

T4RUPT  PROGRAM 

E,T 

DOWNLINK  LISTS 

D 

FRESH  START  AND  RESTART 

G 

RESTART  TABLES 

B 

S XTM  ARK 

I 

EXTENDED  VERBS 

C 

PINBALL  NOUN  TABLES 

H 

KOOLADE 

CSM  GEOMETRY 

J 

IMU  COMPENSATION  PACKAGE 

E,  U 

PINBALL  GAME  BUTTONS  AND  LIGHTS 

H 

P60, P62 

K 

ANGLFIND 

K 

GIMBAL  LOCK  AVOIDANCE 

K 

KALCMANU  STEERING 

K 

SYSTEM  TEST  STANDARD  LEAD  INS 

T 

IMU  CALIBRATION  AND  ALIGNMENT 

U 

GROUND  TRACKING  DETERMINATION 

PROGRAM  - P21-P29 

L 

P34-P35,  P74-P75 

M 

SMOOCH 

P31 

C 

P76 

M 

P80 

C 

STABLE  ORBIT  - P38-P39 

M 

Pll 

N 

TP I SEARCH 

M 

P20-P25 

L 

P 3 0 » P3  7 

M 

P40-P47 

N 

P51-P53 

P 

LUNAR  AND  SOLAR  EPHEMERI DES  SUBROUTINES 

R 

PANDORA 

P61-P67 

A 

SERVICER207 

I 

ENTRY  LEXICON 

comment 

REENTRY  CONTROL 

A 

CM  BODY  ATTITUDE 

K 

P37, P70 

M 

S-BAND  ANTENA  FOR  CM 

C 

. 

LUNAR  LANDMARK  SELECTION  FOR  CM 

I 

I 


Table  4-8  FUNCTIONAL  CATEGORIES  APPLIED  TO  COLOSSUS  237  (corn.) 


Sub- 

routine 


DAPCSM 


ATRAP 


Log  Section 


TVCINITIALIZE 
P 1 5 

TVCEXECUTIVE 
TVCMASSPROP 
TVCRESTARTS 
TVCDAPS 
TVCSTROKETEST 
TVCRCI I DA  P 
TVCGENBFI LTERS 
MYSUBS 

RCS-CSM  DIGITAL  AUTOPILOT 
AUTOMATIC  MANEUVERS 
RCS-CSM  DAP  EXECUTIVE  PROGRAMS 
IFT  SELECTION  LOGIC 
CM  ENTRY  DIGITAL  AUTOPILOT 


DOWN-TELEMETRY  PROGRAM 
INTER-BANK  COMMUNICATION 
INTERPRETER 

FIXED-FIXED  CONSTANT  POOL 
INTERPRETIVE  CONSTANTS 
SINGLE  PRECISION  SUBROUTINES 
EXECUTIVE 
WAITLIST 

LATITUDE  LONGITUDE  SUBROUTINES 
PLANETARY  INERTIAL  ORIENTATION 
MEASUREMENT  INCORPORATION 
CONIC  SUBROUTINES 
INTEGRATION  INITIALIZATION 
ORBITAL  INTEGRATION 
INFLIGHT  ALIGNMENT  ROUTINES 
POWERED  FLIGHT  SUBROUTINES 
TIME  OF  FREE  FALL 
STAR  TABLES 

AGC  BLOCK  TWO  SELF-CHECK 

PHASE  TABLE  MAINTENANCE 

RESTARTS  ROUTINE 

IMU  MODE  SWITCHING  ROUTINES 

KEYRUPT,  UPRUPT 

DISPLAY  INTERFACE  ROUTINES 

SERVICE  ROUTINES 

ALARM  AND  ABORT 

UPDATE  PROGRAM 

RTB  OP  CODES 


Functional 
Category 
( coded ) 


' 


Table  4-9  FUNCTIONAL  CATEGORIES  APPLIED  TO  LUMINARY  98 


Log  Section 


ERASABLE  ASSIGNMENTS 


INTERRUPT  LEAD  INS 
T4RUPT  PROGRAM 
RCS  FAILURE  MONITOR 
DOWNLINK  LISTS 
AGS  INITIALIZATION 
FRESH  START  AND  RESTART 
RESTART  TABLES 
AOTMARK 

EXTENDED  VERBS 
PINBALL  NOUN  TABLES 
jEMONAI  D LEM  GEOMETRY 

IMU  COMPENSATION  PACKAGE 
R 63 

ATTITUDE  MANEUVER  ROUTINE 
GIMBAL  LOCK  AVOIDANCE 
KALCMANU  STEERING 
SYSTEM  TEST  STANDARD  LEAD  INS 
IMU  PERFORMANCE  TESTS  2 
IMU  PERFORMANCE  TESTS  4 
PINBALL  GAMES  BUTTONS  AND  LIGHTS 
R 60 , R 62 

S-BAND  ANTENNA  FOR  LM 


Functional 
Category 
( coded ) 


— 

LEMP2  OS 

RADAR  LEADIN  ROUTINES 
P20-P25 

LEMP30S 

P30, P37 

P32-P35,  P72-P75 

GENERAL  LAMBERT  AIM  POINT  GUIDANCE 

KISSING 

GROUND  TRACKING  DETERMINATION  PROGRAM  - P2 1 
P34-P35,  P74-P75 
R 31 
P76 
R 30 

STABLE  ORBIT  - P38-P39 

FLY 

BURN,  BABY,  BURN  — MASTER  IGNITION  ROUTINE 
P40-P4  7 

THE  LUNAR  LANDING 

THROTTLE  CONTROL  ROUTINES 

LUNAR  LANDING  GUIDANCE  EQUATIONS 

Table  4-9  FUNCTIONAL  CATEGORIES  APPLIED  TO  LUMINARY  98  (cont.) 
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Sub- 

Log  Section 

Functional 

routine 

Category 

P70-P71 

N 

P12 

N 

FLY 

ASCENT  GUIDANCE 

0 

SERVICER 

I 

LANDING  ANALOG  DISPLAYS 

C 

FINDCDUP  - GU I DAP  INTERFACE 

0 

LEMP5  0S 

P51-P53 

p 

LUNAR  AND  SOLAR  EPHEMERIDES  SUBROUTINES 

R 

DOWN-TELEMETRY  PROGRAM 

D 

INTER-BANK  COMMUNICATION 

F 

INTERPRETER 

Q 

FIXED-FIXED  CONSTANT  POOL 

all 

INTERPRETIVE  CONSTANTS 

all 

SINGLE  PRECISION  SUBROUTINES 

R 

EXECUTIVE 

F 

WAITLIST 

F 

LATITUDE  LONGITUDE  SUBROUTINES 

J 

PLANETARY  INERTIAL  ORIENTATION 

J 

MEASUREMENT  INCORPORATION 

I 

CONIC  SUBROUTINES 

I 

INTEGRATION  INITIALIZATION 

I 

SKIPPER 

ORBITAL  INTEGRATION 

I 

INFLIGHT  ALIGNMENT  ROUTINES 

P 

POWERED  FLIGHT  SUBROUTINES 

N 

TIME  OF  FREE  FALL 

M 

AGC  BLOCK  TWO  SELF-CHECK 

T 

PHASE  TABLE  MAINTENANCE 

B 

RESTARTS  ROUTINE 

B 

IMU  MODE  SWITCHING  ROUTINES 

U 

KEYRUPT,  UPRUPT 

D 

DISPLAY  INTERFACE  ROUTINES 

H 

SERVICE  ROUTINES 

S 

ALARM  AND  ABORT 

B 

UPDATE  PROGRAM 

D 

RTB  OP  CODES 

S 

T6-RUPT  PROGRAMS 

A 

DAP  INTERFACE  SUBROUTINES 

A 

DAP IDLER  PROGRAM 

A 

P-AXIS  RCS  AUTOPILOT 

A 

LMDAP 

Q-R  AXIS 

A 

KALMAN  FILTER 

R 

TRIM  GIMBAL  CONTROL  SYSTEM 

A 

AOSTASK  AND  AOSJOB 

A 

SPS  BACK-UP  RCS  CONTROL 

A 

4. 2. 5. 3 LANGU  3ES  USED 


The  language  used  on  this  project  was  assembly  language, 
with  interpretive  code  interspersed  throughout  an  assembly 
language  program. 

Typically  interpretive  code  and  machine  instructions,  as 
well  as  data,  are  interspersed  throughout  a log  section,  sub- 
routine or  functional  category.  Thus  there  would  not  likely  be 
a log  section  consisting  only  of  interpretive  code. 

The  interpretive  language  being  used  is  primarily  oriented 
toward  doing  three  kinds  of  arithmetic,  1)  one  that  operates  on 
28  bits  plus  sign  fixed  point  scalar  numbers,  2)  one  that  oper- 
ates on  42  bits  plus  sign  fixed  point  scalar  numbers,  and  3) 
the  third  that  operates  on  three  element  vectors,  each  of  whose 
elements  is  a 28  bit  plus  sign  fixed  point  scalar  number.  The 
vector  arithmetic  includes  provision  for  multiplying  three  ele- 
ment vectors  and  3 by  3 matrices.  Even  though  this  language  is 
thus  considerably  more  powerful  than  a typical  assembly  lang- 
uage, its  form  (syntax)  is  that  of  an  assembly  language  rather 
than  of  a higher  order  language.  For  a fuller  description  of 
the  language  see  Appendix  A.  Use  of  the  interpretive  language 
instead  of  assembly  language  (or  machine  code)  generally  saves 
storage  space  in  the  AGC  memory  at  the  expense  of  speed  of 
execution. 
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SECTION  5 
DATA  SUMMARIES 

The  data  described  in  Section  4 have  been  summarized  and 
presented  i i this  section  in  tabular  and  graphic  form-  The 
following  observations  are  made  from  examining  these  summaries 
with  a knowledge  of  the  software  development. 

5.1  PATTERNS  OF  FLIGHT  SOFTWARE  DEVELOPMENT 

The  general  pattern  over  time  of  the  number  of  software 
modifications  accurately  reflects  the  history  of  program 
development.  Figure  5-1  shows  that  the  periods  of  greatest 

activity  would  be  expected  before  the  Apollo  9 flight  and 
before  the  Apollo  15  fliqht, since  a lonq  lead  development  time  was 
scheduled  for  those  flights  and  they  were  developed  in  parallel 
with  others.  The  data  bear  this  out  (Figures  5-2,  5-3,  5-4; 
Tables  5-1,  5-2,  5-3,  5-4).  In  mid-1970  a new  flurry  of 

activity  followed  the  release  of  Apollo  14,  reflecting  the  fact 
that  a large  number  of  new  capabilities  were  specified  for  the 
Apollo  15  flight-  The  increased  modification  activity  shown 
for  the  Apollo  15  flight  reflects  the  space  saving  activity 
that  took  place  to  enable  the  implementa tion  of  those  major 
impr  ovements. 

At  least  two  ropes  were  under  development  at  all  times. 
Throughout  1967  and  for  a period  in  1969  and  1970,  three  or  more 
were  being  developed  simultaneously.  Many  of  the  modifications 
tabulated  for  these  periods,  therefore,  are  multiple 
implementations  of  the  same  change. 

Apollo  9 was  the  first  joint  flight  with  the  Lunar  Module; 
not  surprisingly,  Table  5-4  shows  that  the  vast  majority  of  mod- 
ifications for  Apollo  9 were  those  made  to  the  Lunar  Module 
program,  Sundance. 

5.2  FUNCTIONAL  CATEGORIES 

Figure  5-5  and  Tables  5-5,  5-6,  and  5-7  show  that  the 

greatest  modification  activity  was  in  mission-oriented 
fuictions;  powered  flight,  navigation,  tracking,  targeting  and 
digital  autopilot  (DAP).  These  functions  were  all  specifically 
related  to  the  lunar  landing  and  the  rendezvous  programs  that 
were  being  newly  developed  during  the  period  covered  by  this 
study. 

5.3  MODIFICATION  ACTIVITY  RELATED  TO  MEMORY  SIZE 

It  was  expected  that  a correlation  would  be  observed 
between  the  size  of  the  functional  categories  and  the  number  of 
modifications  to  them.  The  data  do  not  exhibit  this,  however, 
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as  shown  in  Figure  5-6  and  Table  5-8. 

It  is  probably  true  that  modification  activity  is  more 
directlv  correlated  to  the  specific  development  taking  placer 
as  discussed  in  Section  5-2,  than  to  memory  size. 

A factor  that  should  be  considered  is  the  method  used  in 
this  study  to  determine  the  sizes  of  the  functions.  The  sizes 

were  taken  directly  from  two  intermediate  programs  that  were 
thought  to  be  representative.  Furthermore,  the  determination 
of  size  and  the  assignment  of  functional  categories  to  the  mod- 
ifications were  done  on  different  bases:  the  sizes  were  based 
on  log  sections,  while  the  assignment  of  a category  to  a modifi- 
cation was  based  on  the  actual  content  of  the  modification 
itself.  Thus  the  usefulness  of  the  figures  on  sizes  of  func- 

tions is  questionable- 

Off-line  development  and  checkout  were  more  common  for 
some  functions  than  for  others.  Data  on  this  off-line  activity 
are  not  available,  but  it  may  well  be  a factor  in  these  statis- 
tics. It  is  known,  for  example,  that  the  digital  autopilot 
(DAP)  programming  group  did  a large  amount  of  off-line  work, 

while  the  powered  flight  programming  group  did  not.  The  data 

show  that  the  percentage  of  modifications  vs.  size  for  these 

two  functions  is  in  direct  opposition,  7.4%  of  the  modifications 
vs.  13%  of  the  memory  size  for  the  DAP,  13.1%  of  the  modifica- 
tions vs.  7.8%  of  memory  size  for  powered  flight. 


5.4  MAJOR  MODIFICATION  ACTIVITY 

It  was  expected  even  prior  to  the  examination  of  any  of 
the  data  summaries  that  large  numbers  of  modifications  would 
fall  into  the  categories  of  logic,  in-house  improvements,  corn- 
pool  definition,  interfaces,  configuration  control  and  user 
requests.  The  nature  of  the  Apollo  project  and  its  computer 
architecture  led  to  this  expectation,  which  is  borne  out  by  the 
data  (see  Figure  5-7,  Tables  5-9,  5-10  and  5-11). 

The  use  of  assembly  language  undoubtedly  contributed  to  the 
preponderance  of  logic  errors;  had  a higher  order  language  been 
used,  the  percentage  of  these  errors  would  no  doubt  have  been 
smaller • 

The  number  of  in-house  improvements  was  expected  to  be,  and 
was,  large.  This  category  included  memory  optimizations  as  an 
ongoing  activity. 

Modifications  to  compool  variables  were  expected  to  be 
large  in  number  because  of  the  time-sharing  of  erasable  memory. 
This  limitation  imposed  continuing  requirements  for  modifica- 
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tions  throughout  the  development- 

Interfaces  are  a traditional  source  of  error;  in  the  Apollo 
project#  the  complexities  of  real-time  interactions  and  the  use 
of  assembly  language  coupled  with  interpretive  language 

exacerbated  the  problems  in  this  area- 

The  fixed  bank  architecture  of  the  computer  required 

specific  code  to  address  remote  banks;  in  addition,  the  memory 
organization  was  changed  frequently  to  accommodate  program 
changes,  both  actual  and  anticipated. 

The  "user  requested"  category  is  actually  a more  significant 
factor  than  is  shown  by  the  number  of  modifications  in  that 

category,  since  more  often  than  not,  a single  change  involved 
one  or  more  large  blocks  of  code.  Data  on  the  size  of  the 

individual  changes  are  not  available. 

It  was  expected  that  the  number  of  computational  errors 

would  be  significant,  due  to  the  large  percentage  of  "number- 
crunching" code  and  especially  because  of  the  fixed  point  scal- 
ing that  was  used-  The  data  show,  however,  that  this  category 
was  relatively  small  in  comparison  to  the  others  discussed 

above . 

5-5  DEVELOPMENT  PHASE  VS.  VERIFICATION  PHASE 

Although  Figure  5-8  and  Table  5-12  indicate  that  most  of 
the  modifications  co  each  rope  were  made  during  the  development 
phase  as  expected,  the  information  presented  is  probably  not 
entirely  accurate*  The  configuration  control  dates  for  Apollo 
flights  7,8,  and  9 are  available  and,  therefore,  the  phase  indi- 
cators for  these  flights  are  reliable.  Phase  indicators  for 

later  flights  were  based  on  the  completion  of  level  4 testing, 
which  approximates  the  configuration  control  dates. 

It  should  be  pointed  out  that  some  errors  were  found  after 
configuration  control  but  were  not  corrected  for  that  flight 
and,  therefore,  will  not  appear  in  the  verification  phase  data; 
instead;  corrections  were  implemented  in  the  development  phase 

of  the  next  flight.  Figure  3-1,  page  23,  shows  that  a signifi- 

cant number  of  known  anomalies  were  allowed  to  remain  in  the 
flight  programs. 

5.6  REFERENCED  DOCUMENTS 


Tables  5-13  and  5-14  illustrate  the  references  in  the  data 
to  supporting  documents-  PCRs,  PCNs,  ACBs  and  Anomaly  Reports. 
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SECTION  6 

SUMMARY  AND  RECOMMENDATIONS 
6.1  NATURE  AND  QUALITY  OF  THE  DATA 


The  set  of  data  provided  by  this  study  is  derived  directly 
from  the  program  modifications.  It  is  a complete  history  of  the 
programs  over  the  four-year  period.  It  includes,  therefore, 
all  changes  made  to  the  programs,  including  error  corrections, 
enhancements  of  capability,  deletions  of  obsolete  capabilities, 
changes  in  mission  requirements,  optimizations  of  either  memory 
or  execution  time,  and  improvements  in  elegance  of  construction. 

It  is  not  quite  correct  to  classify  these  data,  then,  as 
an  "error  history",  since  a larger  portion  of  the  items  were  in 
the  nature  of  improvements  rather  than  corrections  to  outright 
mistakes.  On  the  other  hand,  had  the  system  been  specified  and 
built  perfectly  the  first  time,  there  would  have  been  no  need 
for  any  modifications  at  all,  except  for  those  that  reflected 
changing  mission  requirements.  Since  the  requirements  imposed 
by  changes  in  mission  were  relatively  few  in  number  (although 
large  in  number  of  lines  of  code)  , it  is  fair  to  say  that  the 
data  set  represents  the  response  to  system  errors,  some  of 
which  were  software  errors  (mistakes),  some  of  which  represent- 
ed a failure  in  specif ica tion  or  design,  and  others  which  repre- 
sent the  fact  that  the  job  was  not  done  perfectly  the  first  time 
and,  therefore,  could  be  improved. 

The  value  of  this  data  set  to  prospective  analysis  should 
be  considered  in  the  light  of  several  factors  unique  to  the 
Apollo  project: 

A.  The  real-time  multi-programmed  aspects  of  the  programs: 

This  led  to  problems  in  data  integrity  and  data 
availability  in  real-time.  In  addition,  the 

varying  levels  of  demand  on  system  resources 
caused  problems  in  timing  and  assignment  of 
priorities 

B.  The  importance  of  the  man/machine  interface: 

The  crew,  through  the  DSKY , was  an  interactive 
user  of  the  software  in  that  approval  was 
required  at  each  major  program  step;  options 
presented  to  the  crew  enabled  selection  of 
alternate  program  paths.  Furthermore,  many 

special  functions  could  be  invoked,  at  any 
time,  by  crew  request. 
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C.  Ultra-high  reliability  requirements: 

This  imposed  a heavy  programming  and  testing 
burden,  especially  for  real-time  restart 
ability  and  error  checking. 

D.  Severe  memory  limitations: 

This  led  to  extensive  modifications  for  memory 
optimiztion,  as  well  as  the  necessity  for  time- 
sharing of  erasable  data  locations. 

E.  The  computer  architecture: 

This  led  to  many  modifications  to  tailor  both 
fixed  and  erasable  code  to  enable  the  correct 
addressing  mode. 

F.  The  era  in  which  the  development  took  place: 

The  strict  methodologies  in  specification  tech- 
niques, design  standards,  documentation  stand- 
ards and  programming  conventions  were  not  in 
general  use  at  the  time. 

6.1.1  RELIABILITY  OF  THE  DATA 

The  initial  recording  of  the  data  was  done  for  the  purposes 
of  management  control  and  visibility  at  the  time;  it  was  not 
anticipated  that  the  records  would  be  used  for  later  statistical 
analysis.  The  format  did  not,  therefore,  always  provide  suf- 
ficient information  for  the  categorization  process  performed  in 
this  study.  Some  items  required  reference  to  other  material  — 
memos,  management  plans,  presentation  material. 

A team  of  experienced  Apollo  programmers  and  engineers  was 
formed  to  collect  and  categorize  the  data;  had  these  special- 
ized personnel  not  been  available,  it  is  doubtful  that  the  job 
could  have  been  done.  Yet  it  is  possible  that  biases  may  have 
been  introduced  due  to  their  personal  involvement  with  the 
original  project. 

The  inherent  subjectivity  of  the  categorization  process 
should  be  emphasized.  Judgement  was  exercised  by  fourteen 
individuals  in  assigning  categories.  The  risk  was  recognized 
early  in  the  process  and  consultations  produced  a general 
approach  to  be  taken  by  all;  nevertheless,  it  is  impossible  to 
estimate  the  extent  of  the  divergence  in  judgement  decisions. 
Also,  it  must  be  recognized  that  inadvertent  errors  were 
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undoubtedly  made;  it  was  impossible  to  check  for  any  but 
mechanical  errors  (examining  the  fields  of  machined  data)  due 
to  the  large  volume  of  data. 

Specific  fields  which  may  not  be  completely  reliable 
include : 


- reference  field;  there  is  no  way  of  assuring  that 
references  were  always  cited  on  the  modification  report 
when  they  applied;  it  is  reasonably  certain,  however, 
chat  the  recorders  included  all  such  references  when 
they  were  cited. 

functional  category,  modification  category,  and 
modification  description  fields:  subjectivity,  as 

discussed  above,  was  a component  in  determining  these 
f ields. 

software  development  phase  field;  due  to  lack  of 
sufficient  data,  the  date  chosen  for  the  verification 
phase  was  estimated  for  the  later  flights  (Apollo  10 
through  Apollo  17). 

- size  fields:  the  estimates  on  which  the  values  of 

these  fields  are  based  are  discussed  in  paragraph 

4.2.4. 

6.1.2  UNAVAILABILITY  OF  DATA 

Certain  items  would  have  been  included  in  the  data  had  they 
been  available.  These  include: 

- the  dace  of  initiation  of  each  error  correction;  the 

date  or  program  revision  of  the  discovery  of  each 
error:  no  information  is  available  on  when  errors  were 

found  or  when  correction  processes  were  begun. 

detailed  information  on  the  number  of  simulation 
runs,  the  termination  conditions  of  tests,  the  amount 
of  computer  time  necessary  to  isolate  each  error. 

- the  size  of  the  modifications. 

traceability  from  one  modification  to  others:  only 

incomplete  information  is  available  to  establish  the 
effect,  in  causation  of  additional  errors,  of 
incorporating  a given  modification- 

6.2  RECOMMENDATIONS 

One  of  the  prime  goals  of  software  research  is  to  achieve 
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greater  software  reliability.  One  promising  approach  toward 
this  goal  is  to  find  methods  of  measuring  and  predicting  the 
reliability  of  a given  software  product/  much  in  the  way  that 
hardware  reliability  is  measured  and  predicted. 

Another  approach  is  to  attack  the  reliability  problem  at 
its  source/  that  is,  to  ask  what  kinds  of  errors  happen  in 
software  production,  to  attempt  to  analyze  their  causes,  and  to 
develop  tools  designed  to  deal  specifically  with  those  aspects 
of  software  production  that  contribute  to  the  creation  of 
errors. 

In  order  to  implement  these  approaches,  large  quantities 
of  error  data  must  be  analyzed.  This  means  that  errors  must  be 
tracked  from  the  very  beginning  of  software  development,  and 
further,  that  they  must  be  recorded  in  a way  that  will  enhance 
analysis  of  their  types  and  causes. 

A study  such  as  this  one  reveals  the  difficulty  of 
compiling  data  from  sources  that  were  originally  designed  for 
other  purposes.  Had  more  analysis  of  the  data  been  included  at 
the  time  it  was  produced,  the  job  of  compiling  the  data  would 
have  been  merely  a clerical  one.  As  it  was,  specialized 
personnel  were  required  to  expend  considerable  effort  in  the 
compilation  and  categorization  process.  Further,  some  data 
that  would  have  been  useful  for  analysis  were  not  available 
(see  paragraph  6.1.2). 

The  first  recommendation,  therefore,  is  that  reporting 
procedures  be  designed  so  that  the  material  collected  during 
development  will  be  useful  for  later  analysis.  Specific 
studies  should  be  conducted  to  determine  just  what  information 
is  relevant  to  error  analysis  and  an  effort  to  compile  for 
several  large  current  projects  should  be  instituted. 

In  practice,  however,  there  is  extreme  difficulty  in 
attempting  to  maintain  error  records  during  project  development. 
The  realities  of  schedules  and  costs  make  it  almost  impossible 
for  project  engineers  and  programmers  to  devote  their  time  to 
anything  but  the  process  of  software  production,  test,  and 
verification. 

Methods  must  be  found,  therefore,  that  can  automate  the 
production  and  collection  of  error  data  without  significantly 
impacting  the  schedule  or  the  cost  of  the  host  project. 

Studies  should  be  conducted  to  investigate  the  feasability 
of  incorporating  error  analysis  and  compilation  of  statistics 
into  existing  tools.  Modern  compilers  already  contain  sophisti- 
cated diagnostics;  the  use  of  these  as  a basis  for  maintaining 
error  statistics  may  be  easily  implemented.  Similar  existing 
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diagnostics  contained  in  simulators  and  other  testing  tools 
could  be  enhanced  to  provide  error  histories  as  well. 

Much  work  has  already  been  done  to  prevent  errors  at  their 
source.  Higher  order  languages  themselves  preempt  many  of  the 

errors  common  to  assembly-coded  programs;  structured  program- 
ming is  generally  recognized  as  a giant  seep  forward  and  docu- 
mentation and  control  techniques  have  been  greatly  improved. 

Still,  there  is  a need  for  a better  understanding  of  the 

roots  of  the  problem,  and  work  toward  this  understanding  should 

be  founded  on  a sound  statistical  base. 

The  creation  of  the  data  base  by  the  sponsor  is  certainly 

a step  in  this  direction. 
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APPENDIX  A 


THE  INTERPRETIVE  LANGUAGE 

The  interpretive  language  used  as  one  of  the  two  languages 
in  which  to  write  the  flight  programs  for  the  Apollo  on-board 
computers  can  well  be  viewed  as  a language  in  which  to  write 
instructions  for  a virtual  machine.  This  machine#  called  an 
interpreter#  has  its  own  instruction  set#  its  own  multi-purpose 
arithmetic  accumulator#  its  own  arithmetic  overflow  indicator# 
its  own  arithmetic  argument  stack  (push-down  list),  its  own  two 
integer  index  registers#  its  two  step  registers  (used  by  the  TIX 
instruction#  which  manipulates  an  index  register),  its  own  60 
switches  (boolean  variables),  its  own  program  (control)  counter# 
its  own  return  address  register,  and  its  memory,  which  is 
essentially  the  same  as  the  memory  of  the  AGC # which  hosts  the 
interpreter.  Each  of  the  interpreter's  instructions  can  be 
viewed  as  consisting  of  an  operation  code  followed  by  0,  1 or  2 

operand  designators. 

Consider  first  a class  of  instructions  whose  operation  code 
can  be  followed  by  either  one  or  no  operand  designator.  The 
operations  invoked  by  these  instructions  are  binary  arithmetic 
operations#  e.g.,  add,  multiply,  subtract  and  divide.  Each 
such  operation  operates  on  two  arguments.  The  first  argument 
is  the  content  of  the  interpreter's  accumulator.  The  second  is 
either  explicitly  designated,  in  which  case  the  operation  code 
is  followed  by  an  operand  designator,  or  not#  in  which  case  the 
operand  designator  is  missing.  If  the  second  argument  is  only 
implicitly  designated,  then  it  is  the  top  of  the  interpreter's 
stack,  and  one  of  the  side  effects  of  the  operation  is  the  pop- 
ping of  the  stack.  If  explicitly  designated  it  can  be  designat- 
ed statically  or  dynamically.  If  designated  statically,  it  is 
designated  by  giving  a fixed  address.  If  designated  dynamical- 
ly# it  is  designated  by  giving  a fixed  address  and  by  designat- 
ing one  of  two  index  registers,  the  argument  being  the  one  at 
che  address  which  results  from  adding  to  the  fixed  address  the 
content  of  the  designated  index  register.  The  result  of  the 
operation  is  returned  to  the  accumulator#  and  the  interpreter's 
overflow  indicator  is  set  if  overflow  occurred  during  the 
operation • 

Consider  next  another  class  of  instructions  whose  operation 
code  is  never  followed  by  an  operand  designator.  The  operations 
invoked  by  these  instructions  are  unary  arithmetic  operations, 
e.g.#  square,  round,  double,  sine  and  complement.  The  opera- 
tions operate  on  one  argument,  the  content  of  the  interpreter's 
accumulator,  with  the  result  of  the  operation  being  returned  to 
the  accumulator,  and  the  interpreter's  overflow  indicator  being 
set  if  overflow  occurred  during  the  operation. 
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Other  instructions  provide  for  testing  the  accumulator  or 

the  overflow  indicator , transferring  data  between  the  interpret- 

er's registers  (e.g.,  accumulator,  index  registers)  and  memory, 
resetting  the  program  counter,  resetting  the  return  address 
register,  manipulating  and  testing  index  registers,  manipulat- 
ing and  testing  switches,  manipulating  the  stack,  and  exiting 
and  conditionally  exiting  from  the  interpreter.  By  an  instruc- 
tion by  which  testing  is  carried  out  is  meant  an  instruction 
which  conditionally  resets  the  program  counter  (i.e.,  resets 
the  program  counter  if  the  test  succeeds) • 

Each  of  certain  store  instructions  with  N operand  designat- 
ors (N=l  or  2)  occupies  N words  of  consecutive  storage.  Any 

other  instruction  with  N operand  designators  (N  = 0,  1 or  2 ) 

requires  N plus  a half  or  N+l  words  of  (not  necessarily  consec- 
utive) storage.  Thus  an  instruction  requires  anywhere  from  a 
half  to  3 words  of  storage.  The  average  amount  of  storage 
required  for  an  instruction  is  around  one  and  a half  words. 

(Of  course  these  figures  do  not  include  the  amount  of  stoiage 

required  to  house  the  interpreter,  which  is  itself  part  of  the 
flight  program  and  takes  up  around  2150  words  of  storage.) 

Interpretive  instructions  are  interpreted  by  the  interpret- 
er only  after  a transfer  of  control  to  the  location  INTERPRET 
is  effected  by  the  AGC  via  a Transfer  Control  (TC)  machine 
language  instruction.  The  word  in  the  location  immediately 
following  the  TC  instruction  is  the  first  to  be  interpreted 

after  invoking  (entering)  the  interpreter-  Transfer  of  control 
from  the  interpreter  to  a designated  real  machine  instruction 
is  effected  by  the  interpreter  via  an  appropriate  exit 

instruction. 
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LOG  SECTIONS  AND  SUBROUTINES 

The  fields  "Log  Section"  and  "Subroutine"  appear  on  the 
modification  reports  (Figures  4-1,  page  27,  and  4-2,  page  28) 
used  as  the  source  of  the  data  of  this  study.  They  are 
explained  below. 

Log  sections,  even  though  recognized  by  the  assembler  as 
program  components,  should  not  be  identified  as  modules.  (3 

examples  of  log  sections  are,  for  subroutine  PANDORA  of 
COLOSSUS  237,  Pll,  TPI  SEARCH,  P20-P25.)  Log  sections  were 

created  during  the  Apollo  development  as  a bookkeeping 
convenience,  to  partition  the  software  into  manageable 
portions.  In  some  cases,  but  not  all,  a given  log  section  was 
assigned  to  a single  group  of  engineers  who  had  responsibility 
for  maintaining  it.  Line  numbering  began  anew  with  each  log 
section;  renumbering  could  be  accomplished  by  an  assembler 

instruction.  A log  section  could  not  be  separately  assembled. 
In  most  cases  (as  can  be  seen  from  Tables  4-7,  page  47  and  4-8, 
page  53,  which  are  described  in  paragraph  4. 2. 5. 1.1)  a log  sec- 
tion was  dedicated  to  a particular  functional  category,  but  was 
not  sufficient  for  carrying  out  the  function  nee  more  than 

one  log  section  was  usually  required  to  carry  function. 

Subroutines,  also  recognized  by  the  assembler  as  program 
components,  again  should  not  be  identified  as  modules.  (4 
examples  of  subroutines  are,  for  COLOSSUS  237,  KILERASE, 

KOOLADE,  SMOOCH,  PANDORA.)  Subroutines  were  also  created  as  a 
bookkeeping  convenience,  to  partition  the  software  into  manage- 
able but  more  inclusive  portions.  A subroutine  could  be  separ- 
ately assembled.  However,  a subroutine  was  seldom  dedicated  to 
carrying  out  a particular  function.  Instead  (as  can  be  seen 
from  Tables  4-8,  page  53,  and  4-9,  page  55),  different  portions 
(log  sections)  of  a subroutine  participated  in  carrying  out  dif- 
ferent functions.  It  should  be  noted  that  this  sense  of 
"subroutine"  is  not  the  same  as  the  common  concept  of  a coded 
module  that  can  be  invoked  by,  and  will  return  to,  another 
module. 
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